Request for shutting down export script to old swgcraft.com

Discussion and suggestions regarding the site and/or forum.

Moderator: Forum Moderator

User avatar
Sobuno
Developer
Posts: 2589
Joined: Sun Mar 25, 2007 2:17 am
Contact:

Post by Sobuno » Sun Sep 02, 2007 7:54 pm

Milla wrote: You can see a strange resource name corruption here:

On swgcraft.com
Idao%, Tatooinian Vegetable Tubers (Entered By SWGCraft.co.uk / Not Verified)
No stats available for this resource (Edit Stats)
» Marked as available 5 hours ago by SWGCraft.co.uk (Unavailable)

On swgcraft.co.uk
Idao%&$@iet, Tatooinian Vegetable Tubers (Entered By Quay / Not Verified) (Unavailable)
No stats available for this resource (Edit Stats)
» Marked as available 6 hours ago by Quay

The correct entry was made by myself on 29th August, 2007 (only my records are available as the resource update has changed the time and date entry):

Idaotwatiet, Tatooinian Vegetable Tubers (Entered By MillaMerani / Not Verified)
DR=9, FL=761, PE=140, OQ=81 (Edit Stats)
» Marked as available 0 minutes ago by MillaMerani (Unavailable / Remove)

However, the Recent Changes entries on swgcraft.com did not reflect the actual changes on the site, as a lot more resources were made unavailable compared to the listing.
Lol, I wondered about this one for a couple of hours then it hit me: The obscure thingy was "twat", so he probably just had his /filter on

Milla
Apprentice Crafter
Posts: 77
Joined: Thu Jul 12, 2007 4:01 pm

Post by Milla » Sun Sep 02, 2007 8:15 pm

Sobuno wrote:
Milla wrote: Also, it seems the mass resource entry window has serious trouble with seemingly normal text input:

swgcraft_start
nodaieium,hardened,5,714,295,859,539,198,569,234
swgcraft_end
A space after swgcraft_end will cause this, I just tested the above in FF and IE and it works
Strange, I could swear that there were no spaces after swgcraft_end (I remember pressing enter immediately after the "d" at one of my attempts. I don't know how the input processing works, are there not any 'trim' functions in PhP to remove any unwanted spaces before or after a string?).

User avatar
Sobuno
Developer
Posts: 2589
Joined: Sun Mar 25, 2007 2:17 am
Contact:

Post by Sobuno » Sun Sep 02, 2007 8:18 pm

Milla wrote:
Sobuno wrote:
Milla wrote: Also, it seems the mass resource entry window has serious trouble with seemingly normal text input:

swgcraft_start
nodaieium,hardened,5,714,295,859,539,198,569,234
swgcraft_end
A space after swgcraft_end will cause this, I just tested the above in FF and IE and it works
Strange, I could swear that there were no spaces after swgcraft_end (I remember pressing enter immediately after the "d" at one of my attempts. I don't know how the input processing works, are there not any 'trim' functions in PhP to remove any unwanted spaces before or after a string?).
Indeed there is, I've just been (and still am) too occupied with other stuff in RL

Added trim() functions now

User avatar
Sobuno
Developer
Posts: 2589
Joined: Sun Mar 25, 2007 2:17 am
Contact:

Post by Sobuno » Sun Sep 02, 2007 8:37 pm

Reenabling despawn sync as I seriously do not think that is where the problem is at

User avatar
Sobuno
Developer
Posts: 2589
Joined: Sun Mar 25, 2007 2:17 am
Contact:

Post by Sobuno » Mon Sep 03, 2007 9:55 pm

Are there still problems?

Milla
Apprentice Crafter
Posts: 77
Joined: Thu Jul 12, 2007 4:01 pm

Post by Milla » Tue Sep 04, 2007 12:43 pm

yes, just checked Dantooine/Farstar

hywopeuiam,high grade,241,872,143

(an existing resource) disappeared here, and was consequently deleted from swgcraft.com.

I tried to reenter the resource here, using the usual method

swgcraft_com
hywopeuiam,high grade,241,872,143
swgcraft_end

Again, no spaces either after 'com' or 'end' still it was broken up into several lines. After a few attempts, the entry was accepted, but the stat fields remained blank. I filled in the empty fields with the correct data, but still the site restored the resource without any stats, as it had been entered by Quay.

Also there is a strange entry on Lok/FarStar:

on swgcraft.com

JazibZero, Fermionic Siliclastic Ore (Entered By SWGCraft.co.uk / Not Verified)
CR=824, DR=939, HR=276, MA=805, OQ=964, SR=913, UT=421 *** (Edit Stats / Verify)
» Marked as available 18 hours ago by SWGCraft.co.uk (Unavailable)

On swgcraft.co.uk

JazibZero, Fermionic Siliclastic Ore (Entered By SWGCraft Parser / Not Verified) (Unavailable)
CR=824, DR=939*, HR=276, MA=805, OQ=964*, SR=913*, UT=421 (Edit Stats / Verify)
» Marked as available 18 hours ago by SWGCraft Parser

The stats are correct, but the resource name is "Bawhoosium".

I wonder how it is possible that SWGcraft parser is the originator of both entries on BOTH sites: this would mean that there was no original living player contributor, but the parser bought an account, logged into the game, and started to survey and sample for resources on its own.

Also, it is strange that the name "JazibZero" was entered instead of the correct "Bawhoosium" name. I checked the member list, and there is no player called "JazibZero".

My idea is the since ...khm... khm... there is still no "recent changes" page on this site, somebody is trying to hack/disrupt the data input because he thinks he can go unpunished and there is no telling (obviously there is, but it requires developer action) who messed up the database. I hope it's not the case.

EDIT: With the "Edit stat" option I have managed to enter the correct stats for "Hywopeuiam"

EDIT2:

From Dantooine/Farstar Chemicals:

iviawio,lubri,536,279
gevvu,dant fiber,213,820,780,537,826

were deleted on swgcraft.com. The usual symptoms: when I re-added them to the planet's resource listing, all properties (resource type, all stat numbers) were marked in bold, which means that these properties were deleted by the export script. These are existing resources, nothing warranted their removal.

I wonder if the 30-day automatic removal has something to do with it... could it be that the script has trouble calculating the 30-day period when the month has changed?

Anyway, since there are tons of issues with the synchronization, I would strongly recommend switching to unidirectional import from swgcraft.com without providing updates to the old sites. It seems that the update process is for some reason bugged and deletes existing resources UNLESS of course human interference is to blame for these mishaps (I have no proof, again, there is no 'recent changes' page developed for this site).

User avatar
Sobuno
Developer
Posts: 2589
Joined: Sun Mar 25, 2007 2:17 am
Contact:

Post by Sobuno » Tue Sep 04, 2007 2:46 pm

Milla wrote: swgcraft_com
hywopeuiam,high grade,241,872,143
swgcraft_end
it's swgcraft_start and swgcraft_end
Milla wrote: Again, no spaces either after 'com' or 'end' still it was broken up into several lines. After a few attempts, the entry was accepted, but the stat fields remained blank. I filled in the empty fields with the correct data, but still the site restored the resource without any stats, as it had been entered by Quay.
You have to use the edit resource interface to edit stats
Milla wrote: on swgcraft.com

JazibZero, Fermionic Siliclastic Ore (Entered By SWGCraft.co.uk / Not Verified)
CR=824, DR=939, HR=276, MA=805, OQ=964, SR=913, UT=421 *** (Edit Stats / Verify)
» Marked as available 18 hours ago by SWGCraft.co.uk (Unavailable)

On swgcraft.co.uk

JazibZero, Fermionic Siliclastic Ore (Entered By SWGCraft Parser / Not Verified) (Unavailable)
CR=824, DR=939*, HR=276, MA=805, OQ=964*, SR=913*, UT=421 (Edit Stats / Verify)
» Marked as available 18 hours ago by SWGCraft Parser

The stats are correct, but the resource name is "Bawhoosium".

I wonder how it is possible that SWGcraft parser is the originator of both entries on BOTH sites: this would mean that there was no original living player contributor, but the parser bought an account, logged into the game, and started to survey and sample for resources on its own.
Odd indeed, I saw it the other night too and tried to track it down, but failed
Milla wrote: My idea is the since ...khm... khm... there is still no "recent changes" page on this site, somebody is trying to hack/disrupt the data input because he thinks he can go unpunished and there is no telling (obviously there is, but it requires developer action) who messed up the database. I hope it's not the case.

EDIT: With the "Edit stat" option I have managed to enter the correct stats for "Hywopeuiam"

EDIT2:

From Dantooine/Farstar Chemicals:

iviawio,lubri,536,279
gevvu,dant fiber,213,820,780,537,826

were deleted on swgcraft.com. The usual symptoms: when I re-added them to the planet's resource listing, all properties (resource type, all stat numbers) were marked in bold, which means that these properties were deleted by the export script. These are existing resources, nothing warranted their removal.

I wonder if the 30-day automatic removal has something to do with it... could it be that the script has trouble calculating the 30-day period when the month has changed?
The 30-day remover works by comparing unix timestamps, so change of month shouldn't be an issue
Anyway, since there are tons of issues with the synchronization, I would strongly recommend switching to unidirectional import from swgcraft.com without providing updates to the old sites. It seems that the update process is for some reason bugged and deletes existing resources UNLESS of course human interference is to blame for these mishaps (I have no proof, again, there is no 'recent changes' page developed for this site).
Would be hard trying to track down the bugs if the sync was switched off

Milla
Apprentice Crafter
Posts: 77
Joined: Thu Jul 12, 2007 4:01 pm

Post by Milla » Tue Sep 04, 2007 5:48 pm

Sobuno wrote: Would be hard trying to track down the bugs if the sync was switched off
I see your point but right now updating the sites are frustrating to a point where I am tempted to give it up entirely.

Believe me, it's no fun spending 3 hours a day correcting deletions and whatnot that results from bad synchronization.

User avatar
Sobuno
Developer
Posts: 2589
Joined: Sun Mar 25, 2007 2:17 am
Contact:

Post by Sobuno » Tue Sep 04, 2007 5:51 pm

Milla wrote:
Sobuno wrote: Would be hard trying to track down the bugs if the sync was switched off
I see your point but right now updating the sites are frustrating to a point where I am tempted to give it up entirely.

Believe me, it's no fun spending 3 hours a day correcting deletions and whatnot that results from bad synchronization.
I see your point, and as a matter of fact, just prior to me reading this post I switched off every synchronization script except for fetching resources (And the 30-day remover)

It's hard testing though as the SWGCraft.com server over there isn't exported

EDIT:

Right, seeing that people started reporting this bug the 2nd of September I am going to go track what code I made just prior to that date

Milla
Apprentice Crafter
Posts: 77
Joined: Thu Jul 12, 2007 4:01 pm

Post by Milla » Tue Sep 04, 2007 6:34 pm

How about a procedure like this one:

- I'll do a full update on this site (minerals, chemicals, gas, water, flora) on an agreed day this week
- With the export script turned off you extract the then-current snapshot of swgcraft.com in XML/CSV
- You run the export script once on swgcraft.co.uk so the old site gets updated
- Then you export again the updated swgcraft.com snapshot and compare it with the snapshot taken before the update with the export script.

This might give you a clue as to what is happening.

Just an idea.

User avatar
Sobuno
Developer
Posts: 2589
Joined: Sun Mar 25, 2007 2:17 am
Contact:

Post by Sobuno » Tue Sep 04, 2007 6:40 pm

Milla wrote:How about a procedure like this one:

- I'll do a full update on this site (minerals, chemicals, gas, water, flora) on an agreed day this week
- With the export script turned off you extract the then-current snapshot of swgcraft.com in XML/CSV
- You run the export script once on swgcraft.co.uk so the old site gets updated
- Then you export again the updated swgcraft.com snapshot and compare it with the snapshot taken before the update with the export script.

This might give you a clue as to what is happening.

Just an idea.
We don't have a specific export script, only a specific import script. The export happens the moment you press submit

Milla
Apprentice Crafter
Posts: 77
Joined: Thu Jul 12, 2007 4:01 pm

Post by Milla » Tue Sep 04, 2007 7:40 pm

well, as to the strange JazibZero entry on Lok/FarStar page:

The guy regged on the old swgcraft.com site yesterday, that sheds some light on the mystery.

Strange, it seems that the import script imports entries from the old site, then, when somebody enters a resource here, the export script deletes some existing entries on swgcraft.com and exports resource entries from here to that site with the SWGCraftparser name tag, without recognizing that they are already existing resources.

Or at least this is what I think had happened.

User avatar
Sobuno
Developer
Posts: 2589
Joined: Sun Mar 25, 2007 2:17 am
Contact:

Post by Sobuno » Tue Sep 04, 2007 8:18 pm

Milla wrote:well, as to the strange JazibZero entry on Lok/FarStar page:

The guy regged on the old swgcraft.com site yesterday, that sheds some light on the mystery.

Strange, it seems that the import script imports entries from the old site, then, when somebody enters a resource here, the export script deletes some existing entries on swgcraft.com and exports resource entries from here to that site with the SWGCraftparser name tag, without recognizing that they are already existing resources.

Or at least this is what I think had happened.
We only emulate what a user can do through the SWGCraft.com site itself. A new resource is parsed to SWGCraft.com this way:
http://swgcraft.com/saveresource.php?ne ... newer1=122
Entering this line the first time will add the resource to the database, the second time will just pop an error message saying that it already exists.

If the resource already is marked in our DB with a swgcraft.com ID, it will use this instead for marking availability:
http://www.swgcraft.com/saveresource.ph ... urceIDHere

For marking unavailable we use:
http://www.swgcraft.com/currentresource ... net=10&va=
We loop the above address for each swgcraft id associated with a specific resource (As to make it unavailable on every planet when marked as unavailable)

For removing a resource we use:
http://www.swgcraft.com/currentresource ... 10&va=$row['swgcraft_id']

The only one of these functions that has recently been implemented is the last one listed, so if any of these are to blame, I'd guess it would somehow be this one


Regarding the JazibZero entry, I see one possibility of how it could have been possible: A user entered the resource, which made it get exported to SWGCraft.com, then removed the resource from here (Seeing that we hadn't imported ID's from SWGCraft.com yet, it wouldn't despawn there). Later on, the importer would run, thus importing a resource entered here to here.

EDIT: Looking further into the JazibZero entry, it gets more odd. The time of entry was 2007-09-03 19:09:55 which is not near the time of the hour that the importer runs. The importer changes the entry time to the entry time of the resource on SWGCraft.com if it is earlier though


I am going to disable the Remove option again for now seeing that I suspect it is the one having issues (Somehow :S)

User avatar
Sobuno
Developer
Posts: 2589
Joined: Sun Mar 25, 2007 2:17 am
Contact:

Post by Sobuno » Wed Sep 05, 2007 7:58 pm

Anyone had any issues since my last post (I still haven't fully enabled SWGCraft.com sync)?

Milla
Apprentice Crafter
Posts: 77
Joined: Thu Jul 12, 2007 4:01 pm

Post by Milla » Fri Sep 07, 2007 1:49 am

Just completed a full update on swgcraft.com.

Let's cross our fingers that the 'remove' option was the culprit.

However, I noted one thing - the import script of this site does not update resources here that have been entered without stats, but are listed with all stats on swgcraft.com.

Post Reply

Who is online

Users browsing this forum: No registered users and 11 guests