There are now four phases:
For the first 3 phases, the `All' word means all the channels in the websources file, or you can name a specific channel in the websources file. The websources file can hold more station names than you use to create a slice. This is to cater for people who make dozens of different slices, some of which contain common stations.
Only the ./fetch_data and ./make_slice phases are mandatory. The ./genre_search and ./add_data phases are optional.
The .guiderc file now allows you to specify the name of every run-time file, so can you do this:
Configdir = ./FilesNumbers = ./Files/numbers
Stations = ./Files/stations
Episodes = ./Files/episodes
Websources = ./Files/websources
The Ebroadcast and Foxtel fetchers have been rewritten. They are now much more robust and can also extract episode titles.
If you have multiple web sources of data, you can fetch from one source and then fetch from another source if the first one fails. You would make two websources files, e.g websources and websources-backup. You would make two config files, e.g .guiderc and .guiderc-backup, and they would be identical except that the first would say:
Websources = ./Files/websourcesand the second one would say
Websources = ./Files/websources-backupNow, you could write a shell script loop that does this:
./fetch_data ABC 0 6
if [ "$?" = "1" ]
then echo Error fetching ABC, trying backup web source
./fetch_data -c .guiderc-backup ABC 0 6
fi
Or, you could just fetch all the web data and try all the backups
./fetch_data All 0 6
if [ "$?" = "1" ]
then echo Error fetching some web data, trying our backups
./fetch_data -c .guiderc-backup All 0 6
fi
The first fetch_data won't make intermediate files unless there was some web data. The second fetch_data will only fetch if there is no web data for a channel, and it won't rebuild the intermediate files unless they don't exist.
The programs database can now be kept either in a file (Files/programs) or in a MySQL database. If you want to keep it as a file, make this symbolic link in the TiVo/ directory:
% cd TiVo; ln -s ProgramDbFile.pm ProgramDb.pmIf you want to keep the database in MySQL, make this symbolic link:
% cd TiVo; ln -s ProgramDbSQL.pm ProgramDb.pmThe SQL database schema is documented in ProgramDbSQL.pm.
If many slices are being made which contain common data, it is pointless to rescan the programs database and the movie ratings database several times for the same program. There is now the optional phase called add_data, which takes the intermediate files in the Data directory, adds in the extra information and stores the new intermediate files in a new directory defined in the configuration file:
Extradir = ./Data2This phase loads the programs database, the trimtitles file, the Movie ratings file and now the Movie directors file.
The file format for the Movie ratings file and the Movie directors file is now different from the IMDB file format, because the IMDB format slows down the parsing. In Misc/ there are two converters that convert the IMDB format into the new format. Once the conversion is done, you need to gzip the converted files and store them in Files/:
Files/directors.tivo.gz and Files/ratings.tivo.gz
make_slice builds a slice for All the stations in the stations file, or just one station, for the list of days given. This is the only phase which will update the episodes file and the numbers file, and so this is the only phase which has the -n option.
The output file name can be specified with two %ds in the configuration file like this:
Slicefile = ./AC02601_%d-%d.sliceThe first %d becomes the TiVo daynumber for the start day. The second %d becomes the TiVo daynumber for the end day. If you don't use %ds in your config file, then your output filename will be the same for all slices.Textfile = ./AC02601_%d-%d.txt
make_slice will first try to get intermediate data from the Data2/ directory (i.e from the output of add_data). But if that fails, it will get it from the Data/ directory, and it will do the adding of genres, stars, directors etc. on the fly.
The stations and websources files have been changed to properly handle timezone shifting between transmitter and receiver.
A typical websources entry now looks like this:
Channel Fetcher Timezone URL information ======================================================== ARNA Foxtel Australia/Sydney ARN # Arena
The timezone is specified as it appears in most Linux or Unix boxes in /usr/share/zoneinfo. This is the timezone of the transmitting station.
Typical stations entries now look like this:
Channel Station-id Timezone ================================================================ ARNA Station/1/7000100 Australia/Melbourne # Arena ARNA Station/1/7000100 DoNotShift # Arena
The timezone is the same format as the websources file. Here it specifies where the receiving TiVo is. In this case it is in Melbourne, so there will be an appropriate timezone shift from Sydney to Melbourne; usually 0 hours except that they change to/from daylight saving at different times so it could be +1 and it could be -1 on some days.
Sometimes the provider of a channel does its own time delay, so that a program appears at the same local time in all timezones. If this is the case, set the timezone to the magic word ``DoNotShift''. This will preserve the original times for all programs on that channel, i.e. no timezone shifting will be done. You would use the second entry as shown above.
You need a config file for each slice which names the output slice name and the stations file for that slice. You need a stations file for each slice. It is a good idea to have a common trimtitles, programs file, episodes file and websources file. For my setup here in Australia, I keep everything in the Files/ directory:
directors.tivo.gz rc-TA07001 stations-NT00806 episodes rc-TA07004 stations-QL04001 genretypes rc-TA07005 stations-QL04002 numbers rc-TA07006 stations-QL04003 ratings.tivo.gz rc-VI03001 stations-QL04004 rc-AC02601 rc-VI03002 stations-QL04005 rc-AC02602 rc-VI03003 stations-QL04006 rc-AC02603 rc-VI03004 stations-SA05001 rc-NS02001 rc-VI03005 stations-SA05003 rc-NS02002 rc-VI03006 stations-SA05004 rc-NS02003 rc-VI03007 stations-SA05005 rc-NS02004 rc-WA06001 stations-SA05006 rc-NS02005 rc-WA06003 stations-TA07001 rc-NS02006 rc-WA06004 stations-TA07004 rc-NS02007 rc-WA06005 stations-TA07005 rc-NT00801 rc-WA06006 stations-TA07006 rc-NT00803 rrc-backup stations-VI03001 rc-NT00804 stations-AC02601 stations-VI03002 rc-NT00805 stations-AC02602 stations-VI03003 rc-NT00806 stations-AC02603 stations-VI03004 rc-QL04001 stations-NS02001 stations-VI03005 rc-QL04002 stations-NS02002 stations-VI03006 rc-QL04003 stations-NS02003 stations-VI03007 rc-QL04004 stations-NS02004 stations-WA06001 rc-QL04005 stations-NS02005 stations-WA06003 rc-QL04006 stations-NS02006 stations-WA06004 rc-SA05001 stations-NS02007 stations-WA06005 rc-SA05003 stations-NT00801 stations-WA06006 rc-SA05004 stations-NT00803 trimtitles rc-SA05005 stations-NT00804 websources rc-SA05006 stations-NT00805 websources-backup
Note that the websources file specifies all the channels I fetch, but websources-backup only has some of the channels, and lists the backup web sources.
For the fetch_data I use -c Files/rc-AC02601 All 0 6 to fetch from the primary websources and -c Files/rrc-backup to fetch from the backup websources.
Then I ./genre_search -c Files/rc-AC02601 All 0 6, but it doesn't matter which config file I use here as long as it points to all the channels, i.e to websources.
I run ./add_data -c Files/rc-AC02601 All 0 6 to build the Data2/ files. Finally I do:
for i in Files/rc-* do ./make_slice -c $i All 0 6 done
to build the slices.
Warren Toomey, January 2004.