Version 6.10 Interface Spec
From Newsbin
TCP Interface
Remote Control interface for Newsbin 6.0
Quade Masters
Introduction
This document defines the mechanism for remote control of Newsbin 6.0 through a TCP interface. The Newsbin TCP interface allows the loading filtering and display of usenet file sets. These sets can be loaded using NZB files, local search, internet search or loading existing header data. The design goal is flexibility and giving the client application sufficient functionality to display it's own interface for control of downloads. The older TCP interface is still in place and won't be affected by the new command. The new command interface is really in parallel to the existing one.
TCP connection
Newsbin binds port 118 on either local loopback or 0.0.0.0 meaning accept connections from anywhere.
[SETTINGS] ExternalConnect=1
Is how the mode is controlled. Setting this value to 1 will tell Newsbin to bind to the "0.0.0.0" meaning accept connections from anything. Newsbin needs to be restarted if the mode is changed.
When the client initially connections, the server sends a greetings message including the current newsbin version.
200 Newsbin server <version>
All commands to Newsbin must end in "\r\n" and all responses lines from the Newsbin server will end with "\r\n". At this point the Newsbin server is ready for commands.
Data Model
Each connection to Newsbin maintains it's own context. Internally there's a storage location for file sets (backing store) and a storage location for a filtered version of the backing store (Display list). Several operations can load the backing store with set data, loading NZB's, local and internet search and loading set date from the existing headers located locally on disk. With no filters, the listing operations will show the raw data in the backing store. Filters can be applied to the backing store data. The filters are used to generate the Display list. The commands to list the data are all applied to the display list and not the backing store. The only thing that changes the state of the backing store is to load new data, replacing the old backing store with a new one.
The download list is shared by all connections to the server. This means operations to the download list on one connection will impact the operations of every other connection. In order to synchronize access, all operations should happen within a lock. It would really be safer if only one connection from the application manipulated the download list at a time. The download list is dynamic, meaning downloading will remove elements from the download list asynchronously so, the client has to be ready for stale state. Again, locking the download list, operating on it, then unlocking is the safest way to process the data.
Command interface
Command: Update
Update tells newsbin to download the latest header for the subscribed and active groups.
Command: Status
The status command returns a list of tab delimited status items in standard list format meaning the list ends with a ".\r\n". Currently this interface reports the current speed, the download folder free space, the data folder free space and the amount of data queued in the download list. There are no error responses. Status should always work.
The format is:
Tag<tab>Numeric value.
| Speed | Current speed in Bit/sec |
| DownloadFolder | Download path free space in bytes |
| DataFolder | Data path free space in bytes. |
| Queued | Size of currently queued items in the download list, in bytes. |
The current design can be easily expanded and the client can pick which status it chooses to respond to.
Command: Auth
If Newsbin is set to authenticate the user, then the authentication command must be sent one time to unlock the connection. The client can determine auth state by the initial response from the server.
A 480 response means the client must authenticate, a 200 response means no authentication is required.
"Auth <Hashed Newsbin key + password>"
To generate the hashed key:
- Copy the Newsbin key to a buffer. - Convert it to uppercase in the buffer - Append the '|' symbol. - Append the password - leave it the same case as entered in newsbin - Calculate the MD5 of the combined string - Convert the resulting hash into the 32 byte hex representation of the binary string. - Send the "AUTH <converted hash>\r\n" to newsbin
The password is set in Newsbin the the Remote Control options. The
Newsbin key is the newsbin key associated with normal registration.
Auth only functions if Newsbin is registered so, trying to
authenticate on an unregistered version will result in a
414 - Auth is enabled but, Newsbin isn't registered
412 - Auth is enabled but, no password is set";
These two errors happens if there aren't enough parameters or if the key and password are incorrect. 401 - Authentication Failed - Auth <Newsbin Key> <Password> 480 - Authentication Failed - Auth <Newsbin Key> <Password>
200 - Newsbin Server <Version>
Means the authentication succeeded
Command: Save
This command saves the current configuration file. There are only 2 possible responses.
200 - Configuration saved 418 - Save failed.
Command: Close
This command will close Newsbin.
200 - Newsbin Exiting
Command: Pause
This command will pause, resume and show pause state in newsbin.
Pause by itself will return the current pause state. Two states are possible.
STATUS_PAUSE_OFF - Download Resumed";
STATUS_PAUSE_ON - Download Paused
Pause On
Pause on will pause Newsbin's download. There is only one possible response
STATUS_PAUSE_ON - Download Paused
Pause Off
Pause Off will resume Newsbin's download. There is only one possible response
STATUS_PAUSE_OFF - Download Resumed
Command: SetVar/GetVar
Newsbin maintains an internal configuration table which can be set and retrieved through the remote control interface. The configuration file is a standard INI file with sections and values
[SETTINGS] MemCacheLimt=100
For example. Any setting can be changed on the fly but, not all settings are read by Newsbin each time so, some settings require a restart to apply.
SetVar <section:name:value> GetVar <section:name>
Are how you can set and retrieve these setting in the configuration file. The "Save" command allows you to make them permenant.
GetVar Settings:MemCacheLimit
SetVar Settings:MemCacheLimit:200
Getvar returns a value in standard list response. 200 prompt + one variable and a ".\r\n" indicating "end of list"
Reponses Are: ERROR_VAR_FORMAT_ERROR - SetVar - Format Problem with section ERROR_NOT_ENOUGH_PARAMS - SetVar Command ERROR_VAR_UNABLE_TO_SET - SetVar - Failed to set variable STATUS_OK - Variable: <variable> set to <value> STATUS_OK - Variable Follows\r\n";
Command: GetSection
The GetSection command is similar to the GetVar command in that it lets you retrieve configuration parameters. It differs in that it'll return a complete listing of a section of the configuration files.
GetSection Settings\r\n
For example will return a list of variable names in the specific section mentioned. The groups for example are stored in a section with no values so, you could retrieve the groups list this way.
GetSection Groups\r\n
Responses are: ERROR_NOT_ENOUGH_PARAMS - GetSection Command ERROR_VAR_VARIABLE_NOT_FOUND - GetSection - Section not Found STATUS_OK - Section List Follows
Command: List
All list command follow the same format both for request and for response. The command is sent to the server and then the server acknowledges the command with either 200 status message meaning the data is to follow or with a 400 status message indicating a problem with the command or missing data. Following the 200 response will be some number of responses ending in "\r\n". The final response indicated end of list with a ".\r\n".
List Groups
Will return a list of all subscribed groups.
List Gogs
Will return a list of all group topics (parents)
List Children
Will return a list of all children of a particular parent topic.
List Children <Parent> is the command format.
List Servers
Will return a list of all servers
Command: Filter
Clear SetMinSize SetMaxSize SetAge SetTextFilter SetProfile EnableFilters HideOld MarkOld AddGroup
Each command can change the state of the display list so, the client application should re-load the listing for display
Filter Clear
Clears the existing size filters and groups.
Filter SetMinSize <bytes>
Sets the minimum size filter for when search or group loading is performed.
Filter SetMaxSize <bytes>
Sets the maximum size filter for when search or group loading is performed.
Filter SetAge <hours>
Sets the maximum age of the sets in the display buffer
Filter SetTextFilter <text>
Sets the current text filter. Text quoting is optional (but recommended). An empty string will clear the text filter.
filter settextfilter "bla" - sets the filter. filter settextfilter "" - clears the filter.
Filter SetProfile <text>
Sets the current filter profile filter. Text quoting is optional (but recommended). An empty string will clear the filter profile.
filter setprofile "bla" - sets the filter profile filter setprofile "" - clears the profile
- Filter EnableFilters <0/1>
Enable the Filters. This can be used instead of clear to enable and disable the filters with a single command without changing the filters state.
Filter HideOld <0/1>
Enable the "hide old" filter
Filter MarkOld <Index>
Tag the files as "Old" both in the display list and on disk
Filter AddGroup
Adds a group or a group of groups if a topic is selected. The topic is expanded into the list of current groups. These groups are user by "Load" and "Search" to specify what groups to look into.
Filter Clear is a way to clear them all. It won't change what's currently in the display buffer. When you change a value with this though and you have existing items in the display buffer, you should assume the existing count and indexes into the display buffer are invalidated so, you have to "Loaded Count" and "Loaded List" to get the current count and items.
Responses Are:
ERROR_NOT_ENOUGH_PARAMS - Filter Command ERROR_RANGE_EXCEEDED - Id Range exceeded STATUS_OK - Succeeded
Command: Search
Search Local <min age> <max age> <keyword>
Searches the groups listed using "AddGroupFilter" or all groups if no groups have been added. It loads each group in turn and only saves the items in memory that match the filter string and size filters. These matched items are added to the display buffer and become accessible using the "Loaded" interface. The min age/max age set how many day in the past to search from. So, a setting of 100 and 0 means "from 100 days ago to 0 days ago( Now)". Only finds records from groups that have actually had headers downloaded.
Search Internet <min age> <max age>
Searches using the search server, the groups listed using "AddGroupFilter" or all groups if no groups have been added. It loads each group in turn and only saves the items in memory that match the filter string and size filters. These matched items are added to the display buffer and become accessible using the "Loaded" interface. The min age/max age set how many day in the past to search from. So, a setting of 100 and 0 means "from 100 days ago to 0 days ago( Now)".
Command: Group
Load:
- Group Load <min age> <max age>
To load the last 10 days of records for instance the min age would be 10 and the max age would be zero. It counts backwards. "Group Load 20 10" would load posts from 20 days ago to 10 days ago.
> Group Delete <set id>
> Group MarkOld <set id>
Command: Loaded
>Loaded Count
>Loaded List <min index> <max index>
>Loaded Sort <column> <ascend>
>Loaded Clear
>Loaded Find <search string>
>Loaded Download <set id> <download path>
>Loaded GetNZB <nMin> <nMax>
Loaded List returns a list of display entries that are currently defined as
Index
Subject
Size
Unix timestamp
From
File count
Par count.
New/Old status
Each field is delimited with a "\t" (tab) except the last which is terminated by the end of line indication "\r\n".
Responses are:
ERROR_NOT_ENOUGH_PARAMS - Group Command
ERROR_GROUP_LIST_NO_GROUP - Group Command Incomplete No groups defined to load
ERROR_AGE_MIN_GREATER_THAN_MAX - Group Command Min age > Max Age
Command: Download
-->Download <id> <--200 filename size -->300 start download. <--file data <--200 ok
Used to request transfer of a file currently listed in the files list. All filenames are assumed to be in UTF-8 format.
Command: Upload
Upload is used to send NZB's to Newsbin through the TCP connection. NZB Uploads are currently capped to 100 megs. There are a number of NZB Upload modes.
Upload nzbbinary <filename> <filesize>
Initiates the upload. All filenames are assumed to be in UTF-8 format. This mode loads the NZB into the internal file store. It can be examines like a loaded group entry using "Loaded Count" and "Loaded List". Downloads can be initiated with the "Loaded Download" command.
> upload nzbbinary <filename> <filesize> < 300 begin upload > Upload.... < 200 <count> upload successful.
The NZB will be uploaded in a binary block. Newsbin will read it one byte at a time until the last byte of the file is transferred. Then it responds with a 200 success response. Newsbin will parse out the NZB and display the count of the sets that are currently in the backing store. You can manipulate the sets with "Loaded List". The filename just needs to be a name. Any existing NZB with the same name will be erased. The NZB's will be stored in the Newsbin temp folder.
ERROR_NOT_ENOUGH_PARAMS ERROR_UPLOAD_MISSING_FILENAME ERROR_UPLOAD_MISSING_FILESIZE ERROR_UPLOAD_UNABLE_TO_SAVE
Upload nzbautoload <filename> <filesize>
Initiates the upload. This mode send the NZB to the existing NZB Autoload folder. The NZB will then be picked up and downloaded using the normal NZB Autoload settings. All filenames are assumed to be in UTF-8 format.
>upload nzbautoload <filename> <filesize> <300 begin upload >Upload.... <200 <count> upload successful.
The NZB will be uploaded in a binary block. Newsbin will read it one byte at a time until the last byte of the file is transferred. Then it responds with a 200 success response. Newsbin save the new NZB to the NZB Autoload folder.
ERROR_NOT_ENOUGH_PARAMS
ERROR_UPLOAD_MISSING_FILENAME
ERROR_UPLOAD_MISSING_FILESIZE
ERROR_UPLOAD_UNABLE_TO_SAVE
Command: Downloads/Wish/Failed
Newsbin stores downloading items in the download list, the wish list can be used to store things temporarily before download and the failed list is where failed downloads go. These lists can be manipulated using the same commands with just the name of the list determing which list to change
downloads count wish count failed count
for instance.
Subcommand: Count
<List Name> Count will return a count of the current elements in the selected list.
STATUS_OK - Count Sets Loaded\r\n";
Subcommand: Lock
<List Name> Lock - is the syntax
The download lists are dynamic. The download list itself shrinks and files download and the user might add or delete entries from any of the lists at any time. Because of that, the "Lock" command allows the remote app to lock the download list, manipulate it then unlock it.
If the app crashes or disconnects. Newsbin will automatically unlock the locked list. It is possible for the list to already be locked. If this happens, the current connection will wait till the list unlocks again. Ideally the application will lock, change then unlock in short order. If the lock is held too long, there's a noticeable stall in the GUI for Newsbin.
Response are: ERROR_NOT_ENOUGH_PARAMS - <List Name> STATUS_OK - <List Name> List Locked
Subcommand: Unlock
<List Name> Unlock - is the syntax
Unlocks the download list again.
Responses Are: ERROR_NOT_ENOUGH_PARAMS - <List Name> STATUS_OK - <List Name> List Locked
Subcommand: Clear
<List Name> Clear
This command will clear the list mentioned. There is no prompting. When the list is cleared the contents can't be recovered though they can be added back into the list.
Responses Are:
STATUS_OK - <List Name> List Locked
Subcommand: List
<List Name> List minindex maxindex
After doing a count, the application has an idea of how many items are in the lists. List can be used to return the string representation of the current items. The min index must be smaller than the max and the indexes are 1's based so, if there are 10 items in the list, to display all 10 the command
"Download List 1 10"
Is used.
The format of the display will be documented later
Responses Are: ERROR_NOT_ENOUGH_PARAMS - <List Name> ERROR_AGE_MIN_GREATER_THAN_MAX - <List Name> List Min Index > Max Index ERROR_RANGE_EXCEEDED - <List Name> List Range exceeded STATUS_OK - <List Name> List Locked
Subcommand: Find
<List Name> Find "string"
The find subcommand can be used to return a listing of just the sets who's subject field matches the search string. The find syntax is standard regular expression. The search string should be quoted if it contains any spaces. There is no upper limit to the number of responses. It's possible that the the while list could be returned depending on the search expression
Responses Are:
ERROR_NOT_ENOUGH_PARAMS - <List Name> ERROR_FIND_INVALID_SEARCH_STRING - <List Name> Find Search string invalid ERROR_AGE_MIN_GREATER_THAN_MAX - <List Name> List Min Index > Max Index ERROR_RANGE_EXCEEDED - <List Name> List Range exceeded STATUS_OK - <List Name> List Locked
Subcommand: Pause
<List Name> Pause <Id>
Using an ID returned in the first columnn of the "List" command, it's possible to pause specific set in the download list.
Responses Are:
ERROR_NOT_ENOUGH_PARAMS - <List Name> ERROR_RANGE_EXCEEDED - <List Name> List Range exceeded ERROR_UNREGISTERED - <List Name> Pause Only available if registered STATUS_OK - <List Name> List Locked
Subcommand: Resume
<List Name> Resume <Id>
Using an ID returned in the first columnn of the "List" command, it's possible to pause specific set in the download list.
Responses Are:
ERROR_NOT_ENOUGH_PARAMS - <List Name> ERROR_RANGE_EXCEEDED - <List Name> List Range exceeded ERROR_UNREGISTERED - <List Name> Pause Only available if registered STATUS_OK - <List Name> List Locked
Constants used with the interface
Response messages from the server
STATUS_OK = 200,
STATUS_BW_LIMITER_ON = 251,
STATUS_BW_LIMITER_OFF = 252,
STATUS_PAUSE_ON = 253,
STATUS_PAUSE_OFF = 254,
STATUS_POST_DATA = 300,
ERROR_UNKNOWN_COMMAND = 400,
ERROR_NOT_ENOUGH_PARAMS = 401,
ERROR_AGE_MIN_GREATER_THAN_MAX = 402, // loading something min age > max age
ERROR_GROUP_LIST_RANGE = 403, // trying to list something out of range
ERROR_RANGE_MIN_GREATER_THAN_MAX = 404, // requested a range and the min >= max;
ERROR_RANGE_EXCEEDED = 405, // requested a range and one of the values is out of range
ERROR_SORT_COLUMN_EXCEEDED = 406, // requested a sort but, the column is out of range
ERROR_GROUP_LIST_NO_GROUP = 407, // trying to load a group that's not in the current sub list
ERROR_UPDATE_GROUP_MISSING = 408, // Update group name empty
ERROR_UPDATE_GROUP_NOT_FOUND = 409, // Update group not found
ERROR_FIND_INVALID_SEARCH_STRING = 410, // Search string isn't a valid RE
ERROR_UPLOAD_MISSING_FILENAME = 411, // When uploading an NZB, the filename seems to be blank
ERROR_UPLOAD_UNABLE_TO_SAVE = 412, // When uploading an NZB, unable to open the destination file
ERROR_AUTH_NOPASSWORD_SET = 413, // Auth is enabled but, no password is set in Newsbin
ERROR_AUTH_NOT_REGISTERED = 414, // Auth is enabled but Newsbin isn't registered
ERROR_UNREGISTERED = 415, // Command disallowed - Newsbin isn't registered
ERROR_VAR_FORMAT_ERROR = 416, // section:variable
ERROR_VAR_VARIABLE_NOT_FOUND = 417, // Var variable not found
ERROR_VAR_UNABLE_TO_SET = 418, // Var failed to set variable
ERROR_UNABLE_TO_SAVE_CONFIG = 419, // Unable to save the current configuration
ERROR_UPLOAD_MISSING_FILESIZE = 420, // When uploading an NZB, No size found
ERROR_UPLOAD_NO_AUTOLOAD = 421, // When uploading an NZB, unable find the autoload folder
ERROR_DOWNLOAD_NOFILE = 430, // When trying to download, the file wasn't found
ERROR_DOWNLOAD_INCOMPLETE = 431, // When trying to download 300 message was never seen
ERROR_FILTER_NOT_FOUND = 440, // When trying to add a filter profile, it wasn't found
ERROR_AUTH_NEED_AUTH = 480, // Auth enabled - request a username and password
ERROR_UNIMPLEMENTED = 500, // Not implemented yet
ERROR_INTERNAL_REMOTE_ERROR = 501 // Internal error
Used for Sorting
static const int C_SUBJECTINDEX = 0; static const int C_STATUSINDEX = 1; static const int C_SIZEINDEX = 2; static const int C_DATEINDEX = 3; static const int C_FROMINDEX = 4; static const int C_GROUPINDEX = 5;
