Wktivoguide - Changes from Version 2 to Version 3

Version 3.1, January 2004. (c) Warren Toomey, BSD license.

New Operation

There are now four phases:

  1. ./fetch_data All 0 6 fetches web data
  2. ./genre_search All 0 6 find genres for unknown programs
  3. ./add_data All 0 6 adds extra information to data files
  4. ./make_slice All 0 6 builds a slice
For all 4 phases, the two numbers are the start and end day relative to today, i.e. 0 is today, 6 is 6 days away. Negative numbers are allowed.

For the first 3 phases, the `All' word means all the channels in the websources file, or you can name a specific channel in the websources file. The websources file can hold more station names than you use to create a slice. This is to cater for people who make dozens of different slices, some of which contain common stations.

Only the ./fetch_data and ./make_slice phases are mandatory. The ./genre_search and ./add_data phases are optional.

Configuration Files

The .guiderc file now allows you to specify the name of every run-time file, so can you do this:

Configdir = ./Files

Numbers = ./Files/numbers

Stations = ./Files/stations

Episodes = ./Files/episodes

Websources = ./Files/websources

Fetching

The Ebroadcast and Foxtel fetchers have been rewritten. They are now much more robust and can also extract episode titles.

If you have multiple web sources of data, you can fetch from one source and then fetch from another source if the first one fails. You would make two websources files, e.g websources and websources-backup. You would make two config files, e.g .guiderc and .guiderc-backup, and they would be identical except that the first would say:

Websources = ./Files/websources
and the second one would say

Websources = ./Files/websources-backup
Now, you could write a shell script loop that does this:

./fetch_data ABC 0 6
if [ "$?" = "1" ]
then echo Error fetching ABC, trying backup web source 
     ./fetch_data -c .guiderc-backup ABC 0 6 
fi

Or, you could just fetch all the web data and try all the backups

./fetch_data All 0 6
if [ "$?" = "1" ]
then echo Error fetching some web data, trying our backups 
     ./fetch_data -c .guiderc-backup All 0 6 
fi

The first fetch_data won't make intermediate files unless there was some web data. The second fetch_data will only fetch if there is no web data for a channel, and it won't rebuild the intermediate files unless they don't exist.

The Programs Database

The programs database can now be kept either in a file (Files/programs) or in a MySQL database. If you want to keep it as a file, make this symbolic link in the TiVo/ directory:

% cd TiVo; ln -s ProgramDbFile.pm ProgramDb.pm
If you want to keep the database in MySQL, make this symbolic link:

% cd TiVo; ln -s ProgramDbSQL.pm ProgramDb.pm
The SQL database schema is documented in ProgramDbSQL.pm.

Add Data

If many slices are being made which contain common data, it is pointless to rescan the programs database and the movie ratings database several times for the same program. There is now the optional phase called add_data, which takes the intermediate files in the Data directory, adds in the extra information and stores the new intermediate files in a new directory defined in the configuration file:

Extradir = ./Data2
This phase loads the programs database, the trimtitles file, the Movie ratings file and now the Movie directors file.

The file format for the Movie ratings file and the Movie directors file is now different from the IMDB file format, because the IMDB format slows down the parsing. In Misc/ there are two converters that convert the IMDB format into the new format. Once the conversion is done, you need to gzip the converted files and store them in Files/:

Files/directors.tivo.gz and Files/ratings.tivo.gz

Make Slices

make_slice builds a slice for All the stations in the stations file, or just one station, for the list of days given. This is the only phase which will update the episodes file and the numbers file, and so this is the only phase which has the -n option.

The output file name can be specified with two %ds in the configuration file like this:

Slicefile = ./AC02601_%d-%d.slice

Textfile = ./AC02601_%d-%d.txt

The first %d becomes the TiVo daynumber for the start day. The second %d becomes the TiVo daynumber for the end day. If you don't use %ds in your config file, then your output filename will be the same for all slices.

make_slice will first try to get intermediate data from the Data2/ directory (i.e from the output of add_data). But if that fails, it will get it from the Data/ directory, and it will do the adding of genres, stars, directors etc. on the fly.

Timezone Handling

The stations and websources files have been changed to properly handle timezone shifting between transmitter and receiver.

A typical websources entry now looks like this:

Channel Fetcher Timezone          URL information
========================================================
ARNA    Foxtel  Australia/Sydney  ARN # Arena

The timezone is specified as it appears in most Linux or Unix boxes in /usr/share/zoneinfo. This is the timezone of the transmitting station.

Typical stations entries now look like this:

Channel Station-id       Timezone
================================================================
ARNA  Station/1/7000100  Australia/Melbourne  # Arena
ARNA  Station/1/7000100  DoNotShift           # Arena

The timezone is the same format as the websources file. Here it specifies where the receiving TiVo is. In this case it is in Melbourne, so there will be an appropriate timezone shift from Sydney to Melbourne; usually 0 hours except that they change to/from daylight saving at different times so it could be +1 and it could be -1 on some days.

Sometimes the provider of a channel does its own time delay, so that a program appears at the same local time in all timezones. If this is the case, set the timezone to the magic word ``DoNotShift''. This will preserve the original times for all programs on that channel, i.e. no timezone shifting will be done. You would use the second entry as shown above.

Setup For Multiple Slices

You need a config file for each slice which names the output slice name and the stations file for that slice. You need a stations file for each slice. It is a good idea to have a common trimtitles, programs file, episodes file and websources file. For my setup here in Australia, I keep everything in the Files/ directory:

 
directors.tivo.gz       rc-TA07001              stations-NT00806
episodes                rc-TA07004              stations-QL04001
genretypes              rc-TA07005              stations-QL04002
numbers                 rc-TA07006              stations-QL04003
ratings.tivo.gz         rc-VI03001              stations-QL04004
rc-AC02601              rc-VI03002              stations-QL04005
rc-AC02602              rc-VI03003              stations-QL04006
rc-AC02603              rc-VI03004              stations-SA05001
rc-NS02001              rc-VI03005              stations-SA05003
rc-NS02002              rc-VI03006              stations-SA05004
rc-NS02003              rc-VI03007              stations-SA05005
rc-NS02004              rc-WA06001              stations-SA05006
rc-NS02005              rc-WA06003              stations-TA07001
rc-NS02006              rc-WA06004              stations-TA07004
rc-NS02007              rc-WA06005              stations-TA07005
rc-NT00801              rc-WA06006              stations-TA07006
rc-NT00803              rrc-backup              stations-VI03001
rc-NT00804              stations-AC02601        stations-VI03002
rc-NT00805              stations-AC02602        stations-VI03003
rc-NT00806              stations-AC02603        stations-VI03004
rc-QL04001              stations-NS02001        stations-VI03005
rc-QL04002              stations-NS02002        stations-VI03006
rc-QL04003              stations-NS02003        stations-VI03007
rc-QL04004              stations-NS02004        stations-WA06001
rc-QL04005              stations-NS02005        stations-WA06003
rc-QL04006              stations-NS02006        stations-WA06004
rc-SA05001              stations-NS02007        stations-WA06005
rc-SA05003              stations-NT00801        stations-WA06006
rc-SA05004              stations-NT00803        trimtitles
rc-SA05005              stations-NT00804        websources
rc-SA05006              stations-NT00805        websources-backup

Note that the websources file specifies all the channels I fetch, but websources-backup only has some of the channels, and lists the backup web sources.

For the fetch_data I use -c Files/rc-AC02601 All 0 6 to fetch from the primary websources and -c Files/rrc-backup to fetch from the backup websources.

Then I ./genre_search -c Files/rc-AC02601 All 0 6, but it doesn't matter which config file I use here as long as it points to all the channels, i.e to websources.

I run ./add_data -c Files/rc-AC02601 All 0 6 to build the Data2/ files. Finally I do:

for i in Files/rc-* 
do ./make_slice -c $i All 0 6
done

to build the slices.

Warren Toomey, January 2004.


File translated from TEX by TTH, version 2.78.
On 15 Jan 2004, 07:48.