Database entropy

[steamhead]

I've just upgraded one of my old Linkstation II MIPs NAS boxes from 4.4 to 5.0.61.

Been working fine for years on 4.4, but with the NAS installers finally reappearing I decided to take the plunge and upgraded (and upgraded my license too).

After a few hours of scratching my head trying to figure out where all the configuration stuff went, I got the server up and running, and then came across the most bizarre problem I have ever seen.

The "By Folder" part of the advanced view showing me just what I had before on 4.4. I tried the Web Browser on my PC, and the PS3 client, all fine. Then I went and had something to eat. When I came back Nothing at all was on the view, when I took the Web Browser interface the Home page showed no links, the PS3 said "There are no titles".

I ran a database rebuild, it's only a small Video only server with 1619 files, and watched the number rise on the Status page, and equally watched them reappear in the Browser and PS3, and then watched as over the next 15 minutes folders gradually disappeared or emptied on the view. Even though folders were disappearing, all 1619 files were still visible on the All Videos view, till eventually that view disappeared.

It is however not a predictable pattern, just a predictable outcome. I've rebuilt my database several times today (a rescan does not bring anything back), and on the current run it's been about 40 minutes since my last rebuild, every folder in my TV section has now disappeared, though the TV folder is still visible, the "All Videos" view show 0 entries, yet my Movie folder is still fully populated at the moment.

Bear in mind this is exactly the same folder structure and files that has worked for years under 4.4, I am at an absolute loss as to what is causing this, and what to do next - though raising an official problem ticket with Twonky will be on the cards if nobody here has any bright ideas.

It's seems to repeat after each rebuild so I should be able to collect plenty of diagnostics.

[mgillespie]

Have you tried 5.0.68? This seems to be better from my experience.

http://www.twonkyforum.com/unsupported/5.0.68/

[steamhead]

Just put 5.0.68 on this morning, and sadly that is doing the exact same thing.

Do a rebuild and the number steadily rises to 1623 (added some more files this morning). The browser can see them for a while and then they disappear gradually.

Any thoughts on where to look for clues, what diagnostics to collect?

Should I be raising an official support call?

[steamhead]

OK, done some more digging. The DB directory seems to build OK (I assume as I don't know for sure what should be in there but it makes some sense), I get directories 11 to 14 with nothing but an empty items.dat file and then directory 15 contains a full replica of my folder structure and an items.dat of 1623KB, presumable co-incidentally 1KB per video file. The contents of which contains all my video file names.

After the entropy kicks in the directory structure and items.dat do NOT appear to change, so that made me think something is probably going wrong with storage that Twonky is using. So I rebooted the entire NAS, not just Twonky, and the server has now been up for 50 minutes without losing a single file - so far. I'll keep you posted.

Sadly I didn't think to do a "ps -ef|grep twonky" before rebooting, but I suspect it might be that the NAS installer might have not dealt with a Twonky 4.4 process that was left running.

Assuming that my server stops losing data now, and the reboot is the fix, would you like me to revert to a clean 4.4 install and upgrade again, see if the entropy re-occurs and capture some diagnostics of what processes were running? Might help someone else in the future!

[steamhead]

Three and a half hours up with no disappearing views. So it looks like the NAS install "missed" something that needed a reboot.

Someone from Twonky let me know if they want me to see if I can do it again with diagnostics, I'm happy to help.

[mgillespie]

Three and a half hours up with no disappearing views. So it looks like the NAS install "missed" something that needed a reboot.

Someone from Twonky let me know if they want me to see if I can do it again with diagnostics, I'm happy to help.

Unfortunatly, the NAS scripts are unsupported and are known to cause errors, the invite is out to the community to maintain them (see the note on the unsupported NAS page). If you have a fix for any of the NAS builds, PM it to me with the NAS model and server version, and i'll make sure the right people get the fixed script for inclusion in the future.

The problem may be that if the scripts don't get updated, and the builds get so broken because of that, the NAS builds may disappear totally.

[steamhead]

I have to say I'm no Linux scripting expert, my platform of speciality is z/OS, but I have found out some more "bits" as the problem came back again. My Linux skills are gradually growing thanks to vague memories of AIX and HP-UX (they're close enough)

First thing I found was that at some point between 4.4 and 5 the naming convention for twonky elements changed from twonkyvision to twonkymedia. So the uninstall wasn't entirely removing 4.4 as I found out when one reboot of the NAS let 4.4 win out and the old version of the server turned up on my PS3.

Another thing that may be causing the problem is that the "working data" is stored in the /var directory, which I know is good practice, but no the Linkstation II the /var directory is on the ram disk which is apparently quite small, and was resulting in things failing. I only noticed this when I tried to create a copy of the /var version of the ini file.

For now I've moved the /var data out to another disk and symbolically linked it back to /var. It seems to be stable again for now.

It would be a real shame to now continue with the NAS scripts, since NAS devices are much more useful as media servers than a PC (due to low power consumption, low size and low noise amongst other things).

But as yet I'm not confident enought to update scripts, hopefully someone else here might have a clue based on my observations.

I did have another problem, in that when I reboot the NAS it looses all the settings, license, shares etc. The ini file in /var appears to be getting updated, but the one in /mnt/hda does not. This MIGHT be a side effect of the space problem, I'll post back findings soon.

[steamhead]

OK, I've learned a lot in the last two days about Twonky, Linux and Linkstations. Hopefully this write up will prove useful to others, but there is a "cry for help" at the bottom (sorry it's a bit long).

So to recap...

After upgrading from 4.4 to 5.0.68 (by way of 5.0.61) on a Linkstation II (MIPS) I ended up with two undesirable symptoms - 1. The contents of my database gradually disappeared over time 2. Every time I rebooted the NAS it lost my License and all other config info. Nothing else had changed on the NAS from the perfectly working 4.4.

What I discovered about the differences between 4.4 and 5.0.68 is that 4.4 stored the database inside the same node as the software (/mnt/hda/twonkymedia), which was on the /dev/hda3 filesystem, which contains the bulk (99.86%) of the 320GB of the device capacity. This worked fine. With 5.0.68 it puts the variable stuff in a separate node under /var (/var/twonkymedia) which I know from a Unix/Linux standards viewpoint is considered good practice. That is until you discover a couple of things about /var on the Linkstation II...

Under the Linkstation II /var is stored within the /dev/ram0 file system which only has 0.003% percent of the NAS available to it. So as the database built and rebuilt it soon ran out of space, which I suspect is the cause of things going missing after a few minutes. My solution to this was to create a symbolic link from /var/twonkymedia to somewhere on /dev/hda3. Which worked fine (no database entropy) till I rebooted the NAS. That's because the next important thing to know about the /var location is it is completely wiped each time which explained why I lost my settings on a NAS reboot as it creates a copy of the ini file in there with all my changed info. This also meant the /var/twonkymedia directory had gone, so twonky promptly created a brand new one, using the default.ini back in /var.

Now luckily the ini file twonky creates in /var has some more info in it over the default one, particularly cachedir=, dbdir=, uploadmusicdir=, uploadpicturedir=, uploadvideodir=, onlinedir= and mediastatisticsdir= all of which were pointing to /var. So I created a new copy of the default ini file, with these keywords pointing to /mnt/hda/twonkymedia based locations (emulating what 4.4 did), copied the /var/twonkymedia stuff into /mnt/hda/twonkymedia (for good measure) and rebooted the NAS. This time MOST of the stuff ended up well away from the /dev/ram0 device, curing the entropy problem, however the following files still found themselves in /var/twonkymedia - byebye, clients.data, db.info, twonkymedia-locations.db and twonkymedia-server.ini. So this still means any settings changes I make get wiped each time the NAS restarts (unless I remember to copy it into my default, which isn't ideal as I lose the ability to revert to default settings)

So my cry for help is simply are there keywords for the ini file that will point these files to be created elsewhere, like you can do with dbdir etc? In fact, even better, is there one keyword to rule them all? So for example if I didn't set dbdir= and onlinedir= could I have set something to make it assume /mnt/hda/twonkymedia as the prefix to /db and /twonkymedia-server-online-data.

Assuming there are such keywords, then I will create a new "default ini" file. I guess this could be distributed with the Linkstation II MIPS installer to stop others having the same problems, and possibly other variants of LinkStation might benefit from it.

[mgillespie]

the keywords in the twonkymedia-server-default.ini are the same as those you will find in /var/twonkymedia/twonkymedia-server.ini

If no config file exists when the server starts for the very first time, it reads the defaults from twonkymedia-server-default.ini to create a new /var/twonkymedia/twonkymedia-server.ini

Thats my understanding of how it works.

As a start, I would try these: (I mount my USB disk as /usb/)

Code: Select all

cachedir=/usb/twonkymedia/db/cache
dbdir=/usb/twonkymedia
uploadmusicdir=/usb/twonkymedia/twonkymedia-server-uploaded-music
uploadpicturedir=/usb/twonkymedia/twonkymedia-server-uploaded-pictures
uploadvideodir=/usb/twonkymedia/twonkymedia-server-uploaded-videos
onlinedir=/usb/twonkymedia/twonkymedia-server-online-data
mediastatisticsdir=/usb/twonkymedia/media-statistics/

As for the remaining files in /var, those are small and unavoidable I believe. If you NAS wipes them, You have 2 options:

1/ hack the mediaserver startup script so it copies in the INI file before starting the mediaserver. The only mandatory file is the INI file telling the server where to store everything else, all other files in /var/twonkymedia will be created at runtime.

2/ mount /var/twonkymedia so it lives on your HDD. rather than in RAM.

The other change that happened between 4.x and 5.x is the PID file for scripts to detect if the server is running has changed. It used to be /var/run/twonky.pid it's now /var/run/mediaserver.pid

[mgillespie]

Actually thinking about it, when you have your correct setup, have you tried saving it as the default INI file, so when the server starts, it sees no files in /var/twonkymedia but does not care because it uses a complete default every time that points to your database files on your HDD?

[steamhead]

Actually thinking about it, when you have your correct setup, have you tried saving it as the default INI file, so when the server starts, it sees no files in /var/twonkymedia but does not care because it uses a complete default every time that points to your database files on your HDD?

Yes I've done that, that was what I meant by this in my previous post...

(unless I remember to copy it into my default, which isn't ideal as I lose the ability to revert to default settings)

What that means is that each time the NAS reboots it gets my "Standard Config" by way of the default but I have to remember if I change my config through the browser that I must copy this to the default.ini or lose it on NAS reboot, of course it also means I can no longer revert to "factory defaults" through the browser without copying a new default.ini into place.

I guess ideally I need to add a script to copy these files into /mnt/hda at NAS shutdown, and back again at NAS startup, before Twonky startup, since you wouldn't want them copying EVERY time time Twonky starts, just when it starts with the NAS.

At least I now have a stable 5.0.68 on a Linkstation II. Once I have my new default.ini AND shutdown scripts completed, I'll post them up here with instructions, but I'm afraid someone else will need to tackle how to integrate that into a NAS install as it's a little beyond me.

On a related front, I'll post requirements for those other files to get keywords to set their location.

Thanks for helping me out. Now I'm off to write some xml views