TMS 5.1.2. updates DataBase forever / Acer H340

[D034599]

Hi there,

after I made really good experiences with an earlier version of TwonkyMedia Server, I decided to buy the new 5.1.2. version.
I tried the WHS installation first, but this was just killing my HomeServerConsole. After some playing around, I could not find anything how to improve the performance of the consolde, that's why I decided to deinstall it again. Somehow Twonky boosted the Memory usage from 50MB to 150MB and made the console pretty much unusable.

Next try is the standalone version. Again, 5.1.2., running on H340 WHS (Windows Server 2003) as service.
The clients are at the moment Iphone and Ipod Touch with PlugPlayer 2.5.0. I am trying to share Music (MP3).
140.000 files (I know... a lot)
I would like to just use the folder display option as ID3 is a mess

I am not going to post the log file as it was more than 200MB when I decided to switch logging off again.
The initial scanning of the MP3's is working pretty fine, it takes around 2 hours to complete the 140.000 files.
Unfortunately it enters a state afterwards, where it keeps using 50% of my CPU forever.
I of course recreated the DB several times and restarted the server repeatedly.

This is what the ServerStatus Page shows (counting up the wpl files ouf course):

Cache Speicher: 44770 KB
Letzte Änderung: Läuft
[+...]Dokumente und Einstellungen\All Users...ungsdaten\TwonkyMedia\db\2951_temp_playlist.wpl
Musikstücke: 139609
Bilder: 0
Videos: 0

The twonky.db has a size of ~93MB

It is sequentially (well, not really, it is taking 1_temp_playlist.wpl, then 10__temp_playlist.wpl, then 11__temp_playlist.wpl etc, you get the idea) taking all the temporary playlist files in \TwonkyMedia\db\ and add's them to the DB. For me, this is around 3500 files. Unfortunately, this takes forever.
Once processed, the wpl files gets deleted from the directory

This is an extract of the log file:
12/31/09 15:23:52:265 upnp_database_add_or_update_playlist - C:\Dokumente und Einstellungen\All Users\Anwendungsdaten\TwonkyMedia\db\2955_temp_playlist.wpl
12/31/09 15:24:39:125 upnp_database_add_or_update_playlist - C:\Dokumente und Einstellungen\All Users\Anwendungsdaten\TwonkyMedia\db\2956_temp_playlist.wpl
12/31/09 15:25:28:062 upnp_database_add_or_update_playlist - C:\Dokumente und Einstellungen\All Users\Anwendungsdaten\TwonkyMedia\db\2957_temp_playlist.wpl

You can see, that it is taking 40 seconds at minimum per wpl. having 3500 wpl files would mean a total scan-time of about 38hours plus the inital 2 hours of plain MP3 scanning.
This is of course a bit too much, even with a huge music collection like mine.

I will attach some hours of playlist idling logs and my ini file.
--------------------------
so there are some questions that arise
1.) What the heck are those temp playlists? Is twonky somehow mirroring the playlists in the shared folders?
2.) is it by intention taking THAT long to parse the files? 3500 does not sound too much and 40hours for it is crazy
3.) do I do something wrong?

Any help is very much appreciated.

Thanks a lot in advance and a great start into 2010,
Dennis

[D034599]

Hey there,

seems to be that I am the only guy with this issues... feels damn lonely
Joking aside.

My scan finished after 44hours. The crazy part is, that it indeed took 42 hours of processing the temp_playlist files in the DB folder. I still do not get the sense behind those files...
Does anybody know why they are created and processes? Is it some kind of workaround for the music structure I have? Is it somehow related to missing playlists in my folders and Twonky has to create new ones to interpret?
Any idea?
I am really not that happy to spend another 40 hours of processing as soon as I am changing my music folder structure (which happens quite often as I am renaming and tagging the new things but still want to be able to access them as soon as I ripped them first hand)

Again, any idea would be great!

[audiokaas2]

Same problem here...

The solution for me was to remove all *.cue/m3u/m3u8 files and rebuild the database.
It's now quick and playlist shows recently added songs etc...

[mcgoo]

same for me.
seems that 5.1.2 is crashing while building the database. 5.1.3 rc-1 does the same. removing cue and m3u files as workaround is not what i like to do. back to 5.1 stable but slow .

[magebarf]

I also have this phenonmenon where it does handling of the XXX_temp_playlist.wpl for quite some time.

I'm running Twonky on a Qnap Ts-459, and I don't know how the processing power of it compares to your machine, but the processing time for each .wpl file is around 5 seconds, so it is probably going a lot quicker on my end.

It might also be because I have a lot of small .m3u files with between 1-10 songs in each... Maybe this is why it take a lot less time on my end? Have you done any experimenting with regards to the size of the playlists?

Well, just to let you know that you're not the only one, and to give the heads up to the Twonky team that this issue exists...

[magebarf]

To fill in:

I am even more sure that it's related to the m3u files now. I pulled the content out of one of the *_temp_playlist.wpl files, and the contents very much indicates that it's due to some conversion or something from a m3u file.

Code: Select all

<?wpl version="1.0"?>
<smil>
<head>
  <meta name="Generator" content="TwonkyMedia"/>
  <meta name="IsNetworkFeed" content="0"/>
  <meta name="IsUserGenerated" content="0"/>
  <meta name="OrigFilename" content="/share/Multimedia/Music/Psy/GMS_Vs_Dimitri_-_Is_Anybody_Out_(Unreleased)-2008-KMx/00_GMS_Vs_Dimitri_-_Is_Anybody_Out_(Unreleased)-2008-KMx.m3u"/>
  <meta name="Type" content="M"/>
  <title>00_GMS_Vs_Dimitri_-_Is_Anybody_Out_(Unreleased)-2008-KMx</title>
</head>
<body>
  <seq>
   <media src="/share/Multimedia/Music/Psy/GMS_Vs_Dimitri_-_Is_Anybody_Out_(Unreleased)-2008-KMx/01_GMS_Vs_Dimitri_-_Is_Anybody_Out_(Unreleased)-KMx.mp3" />
  </seq>
</body>
</smil>

While doing this the process hogs basically all of the processing power it can, while still being marked as in a "sleep" state... Not sure what they mean exactly, but this is what top shows in the process list:

Code: Select all

Load average: 1.18, 1.11, 1.09    (State: S=sleeping R=running, W=waiting)

  PID USER     STATUS   RSS  PPID %CPU %MEM COMMAND
 2859 admin    S        21M  4595 99.9  2.1 twonkymediaserv

D034599, that machine you're running on, is it a dual core CPU in it? I'm guessing that your 50% CPU usage is due to that.

[nathan_h]

Checking in to see whether you have found a solution?

[Levinson]

Same issue on QNAP NAS TS-419 running TwonkyMedia v5.13.

I'am a registrated user of TwonkyMedia server 5.x aswell 4.x.
I run TwonkyMedia on several PC's and NAS servers

My database has 40000 FLAC tracks (4000 albums) the database scan of Twonky Media took 1:15h (comparable with version 4.4.18) however it is still busy doing something with the playlist .wpl . After 40Hours the database built is still not finish but in progress!! I call it it a serious issue, Has someone booked progress to get around it?

Does someone know the reason of this .wpl "feature", older version 4.4.18 didn't had the problem.
** Never change the winning team, 4.4.xx was great **

Thanks
Wim B.

[Saxfusion]

Experiencing a similar issue on ubuntu-server with TMS 5.1.3. For safety i did only choose scan-type 0 (to scan ONCE)
Only choose around 10000 video (well mkv and sort is not recognized anyhow) and audio files. File-types on that volumes are mp3, flac, m3u, cue, zip, rar, tgz, tar. Maybe others.
As of now it is "in progress" since nearly 7 days while the total number of files found has been done within 1-2 hours.
Within /var/twonkymedia/db/ the .wpl files keep piling up. Right now i have around 1.6 million .wpl files with a total of 6.4 Gb.
The ones i picked hold mp3 and m3u references.

Strange thing is that more than three samples i took have the same content.

/var/twonkymedia/db/50000_temp_playlist.wpl
/var/twonkymedia/db/10000_temp_playlist.wpl
/var/twonkymedia/db/100000_temp_playlist.wpl

[code]
<?wpl version="1.0"?>
<smil>
<head>
<meta name="Generator" content="TwonkyMedia"/>
<meta name="IsNetworkFeed" content="0"/>
<meta name="IsUserGenerated" content="0"/>
<meta name="OrigFilename" content="/media/music/lucky/2008/200810/SOMETHING.m3u"/>
<meta name="Type" content="M"/>
<title>SOMETITLE</title>
</head>
<body>
<seq>
<media src="/media/music/lucky/2008/200810/SOMETHING_1.mp3" />
<media src="/media/music/lucky/2008/200810/SOMETHING_2.mp3" />
<media src="/media/music/lucky/2008/200810/SOMETHING_3.mp3" />
<media src="/media/music/lucky/2008/200810/SOMETHING_4.mp3" />
<media src="/media/music/lucky/2008/200810/SOMETHING_5.mp3" />
</seq>
</body>
</smil>
[/code]

I did take even more samples to make sure that there are files with other content though, and there is, luckily.
With flac files the body of the xml is empty.

Something is really going wrong but i can't find out what. If this weren't for a BETA version (assuming 5.1.3-dev display), which is not marked on the download page as such, and if i would have no other means to watch my media i really would start to want my money back. But so i just hope it gets fixed soon...

[D034599]

Hey there everybody,

after not really succeeding to solve the issue on 5.1.2 i decided to pause my efforts a bit until a new version comes out.
I just downloaded the TMS 6 Beta and we still have the same behavior there
I guess that's it for me, testing some other media-servers now.

Too bad, I really liked Twonky so far and would love to stay with it, but this is just too damn slow
I even did not find any way to manually exclude the m3u files from processing (which would have been an easy fix).

@Twonky Dev: I gave you a chance to fix it in a later build, but keeping it even in 6 makes me a bit sad...

//Dennis

[Parnic]

Has there not been any resolution to this issue? I just purchased and installed TwonkyServer 6 on my linux server and it has been generating these .wpl files for over an hour now. Is there a way to disable this feature?

[Twonky_Rick]

Thanks for reporting this. I've asked our lead developer to look into it.

- Rick

[Parnic]

Thanks! Almost all of my 1000+ albums have their own m3u playlist, and it looks like the server is trying to generate a .wpl for each .m3u. I'd love if the temp_playlist feature was explained (what is it for?) and an option was added to disable it.

[Twonky_Rick]

Here is the response I received. Please don't shoot the messenger.

The thread started 2009, the last relevant messages were from June 2010. This issue only occurs with TwonkyServer 5.1.2 which is no longer supported.

You may want to check with your manufacturer and see if they have a software update available. If you're seeing this issue with TS 6.0, please let me know.

- Rick

[Parnic]

I'm seeing the issue with TS 6.0, this was just the first thread that turned up in a search for the problem I'm having. Here are other threads complaining about the issue:
http://www.twonkyforum.com/viewtopic.php?f=2&t=8596 (the suggestion listed here does not work)
http://www.twonkyforum.com/viewtopic.php?f=2&t=7396
http://www.twonkyforum.com/viewtopic.ph ... emp#p32660 (removing m3u files is not something I'm willing to do)

etc.

I can start a new thread complaining about the issue on TS 6 if that will appease the developer .

[Twonky_Rick]

No need. Thanks for posting all of the related links. That will help us.

I'll try to get a ticket opened up for you on this issue.

- Rick

[Twonky_Rick]

Update:

Our Berlin office has asked their testers to look into this. If they can reproduce this, they will create a bug report for this. Once that's done, issues like this are normally fixed -- although it's unclear when this will happen.

- Rick

[Parnic]

Thanks. Note that the reproduction steps most likely revolve around having large numbers of .m3u files in the collection being scanned.

[Twonky_Rick]

Can you give me an idea how many M3U files you have?

- Rick

[Parnic]

I show 858 .m3u files in my music collection.

[Twonky_Rick]

Thanks. This will hopefully help us to reproduce this issue.

- Rick

[MikaelK]

This is still an issue with 6.0.39 running on a DLink DNS-323

[Parnic]

This is still an issue with 6.0.39 running on a DLink DNS-323

And 7.0.7, too .

[Twonky_Rick]

Could both of you temporarily remove or hide your M3U files and let me know if it helps? If it does, we will know this is the cause and I'll make sure they test with a large number of m3u files.

- Rick

[Parnic]

Could both of you temporarily remove or hide your M3U files and let me know if it helps? If it does, we will know this is the cause and I'll make sure they test with a large number of m3u files.

- Rick

I'll see what I can do, but there's not really a good way to 'hide' them or bulk move them away and then back to their proper folders later (that I know of).

[Twonky_Rick]

Another option might be for you to Zip the M3U files up and send them to me. If that's easier to do.

e-mail: tmm-beta@pv.com

- Rick

[fluXXXion]

I updated my QNAP TS-419P+ to TwonkySever 7.0.9 nine days ago.
The creation of these temp playlist files is running since then, which means a very high CPU load over a very long time.
Could you finally please stop this? I do not need these temporary playlists at all, but they keep bothering me since I am using TwonkyServer. In previous versions it took at least 2 days, but 9 (nine!) days at minimum is absolutely inaccepatble.
I would even accept a workaround.

[phibertron]

One option to take

is to first look at this

A solution for Twonky Server 7.x nas startup scanning
http://www.twonkyforum.com/viewtopic.ph ... artup+scan

and then to do this

move all your media out of the twonky media share
and slowly add it back a chunk at a time

this would allow for the cpu and you to deal with the initial build up of the db
once all the media is added in
the boot up/ scanning issues should be ok

[fluXXXion]

And this has to be done after every reboot? After every restart of TwonkyServer due to a change in the settings? Sorry, this solution is unacceptable.

What's the problem in making this an option? As far as I can see this issue is bothering many people...

EDIT: Running 6.0.39 now which runs at ~40 % CPU load while generating the playlists, and it finished after 2,5 hours which is far better then infinitely...

[phibertron]

first off, I dont work for twonky, I just do what I can

And this has to be done after every reboot?
After every restart of TwonkyServer due to a change in the settings?
Sorry, this solution is unacceptable.

the solution I proposed, is one that give the best of of both scenarios, if using 7,x
the solution is not possible in previous version, only in 7.x

What's the problem in making this an option?
As far as I can see this issue is bothering many people...

in 7.x they did make it an option
the catch is, it didnt allow for new media to be detected
which is why I came up with the solution that I did for 7.x

EDIT: Running 6.0.39 now which runs at ~40 % CPU load while generating the playlists, and it finished after 2,5 hours which is far better then infinitely...

thats good to hear, 6.0.39 seems to be a pretty solid build, from what I have seen

[fluXXXion]

EDIT: Running 6.0.39 now which runs at ~40 % CPU load while generating the playlists, and it finished after 2,5 hours which is far better then infinitely...

thats good to hear, 6.0.39 seems to be a pretty solid build, from what I have seen

Well, finally this version was cluttering up my ramdisk space up to 0% free space. Maybe you can imagine what kind of effects this can have.

So I am still waiting for a version without this bug.

[phibertron]

I would take a look at 6.0.38 or 6.037
or 7.0.11 or 7.0.12