Basilisk current data wiped?

[Slyvampy]

The parser hasnt changed at all. I will look into this, the job that collects information from GH is once an hour. I have increased this to every two hours to see if it resolves the issue. (This might be as the job could be taking longer than one hour to complete - there are that many resources to check).

[divory]

didn't help parser is still wiping all data.

[Savacc]

I thought we were only drawing data from GH once per day as they only update once per day and our hourly enquiries were overloading their system?

Also Divory is correct. The problem with Basilisk's current resources is still there. It only lists the "new resources" that spawn since the last time the parser runs and marks all other resources as "unavailable" even though they still have many days yet to go.

[Slyvampy]

Ok, I have changed it to twice a day see if that makes a difference. I am confused as the the file was changed was 22nd of July 2016. Nothing has changed our end, unless GH changed anything we are not aware of.

[Savacc]

Yesterday and today's resources are there, for Basilisk's current resources. I guess we wait and see what happens tomorrow?

[Savacc]

We've got resources from three days now. Give us two more weeks of this and we will be back to normal

[divory]

should we mark available the stuff that has been inappropriately mark unavailable?

[Savacc]

Can you do that? If you can, sure, it's worth a try. If it takes an admin, let me know and I'll do it, or it least some resources to track.

Thing is, this is not how it is supposed to work. we should get a full report of all that GH has, each day. If it is not on GH's feed, it gets marked "unavailable". Slyvampy seems convinced it is a fault on GH's feed. He could be right. I have not read anything on the Emu forums saying anyone besides us has noticed this problem. That tends to make me think it is our error, but maybe not. Maybe we are the only ones using GH's feed and the only ones that will notice?

We are really waiting for Sobuno to look at this. He responded to an email with a request for more info, but that was week ago. He last logged on here a week ago Sat. So real life must be keeping him away

[divory]

the parser data seems to be 12 hours behind Galaxy Harvester. Also some spawns like the new phrik alum says its suppose to be available on talus and its not. Otherwise aside from 293 listed and SWGAide showing 291 and 12 hours off it seems to be mostly accurate. Anyone else noticing discrepancies?

[Sobuno]

Our GH Parser has not changed since 2016-07-22 as Slyvampy correctly has identified.

What follows is an outline of the logic it performs:

Preparation:

1. Find all servers that have a GH ID associated
2. Find all planets and make some changes to match the naming on GH (Space becomes underscore, the 4 in Yavin 4 becomes IV)
3. Find all resources previously imported that are available and keep a list of them (Called the "missing" list - At the end, all resources still listed here are marked unavailable).
4. Generate a list of files to fetch from Galaxy Harvester on the format:

   Code: Select all

   http://galaxyharvester.net/exports/current<GH id>.csv

A file looks like this:

Code: Select all

"name","galaxy_id","galaxy_name","enter_date","type_id","type_name","group_id","CR","CD","DR","FL","HR","MA","PE","OQ","SR","UT","ER","unavailable_date","planets"
"heapefe",14,"Basilisk","Mon, 29 Oct 2018 03:58:45 -0800","aluminum_agrinium","Agrinium Aluminum","aluminum",110,491,253,0,349,436,0,633,371,457,0,,"Corellia|Dantooine|Lok|Naboo|Rori|Talus|Tatooine|Yavin 4"
"meagio",14,"Basilisk","Tue, 30 Oct 2018 04:17:55 -0800","aluminum_agrinium","Agrinium Aluminum","aluminum",132,428,302,0,314,538,0,4,516,485,0,,"Dantooine|Dathomir|Lok|Naboo|Rori|Talus|Tatooine|Yavin 4"
"onaic",14,"Basilisk","Sat, 27 Oct 2018 04:03:18 -0800","iron_axidite","Axidite Iron","iron",315,192,584,0,609,135,0,11,642,542,0,,"Rori|Tatooine"
...
...
...

For each file fetched:

1. Retrieve file
2. Delete first line (It contains the column names, e.g. name, type, stats)
3. For each following line, Check if a resource with the given name for that line already exists on the server:
   1. If it doesn't exist, check if the resource is planet-specific and add it to the appropriate planet. If not planet-specific, parse the planet names at the end of the line and add it to the planets we recognize.
   2. If it does exist, some different logic applies:
      1. Remove the resource from the "missing" list if present
      2. Add stats that are missing on a resource if GH has stats for fields we are blank on.
      3. Mark resource as imported
      4. If we already have the resource marked as available, we parse GH's list of planets it is available on and mark it available there as well. If we do not already have the resource marked as available, we do no such thing. This is in order to trust our own availability data more than GH's, otherwise any changes made to Availability on SWGCraft would be reverted the next time the Parser runs.

And finally, at the end, every resource still on the "missing" list will be de-spawned.

[Sobuno]

Looking at the logs it generates, I do see that it sometimes had a fatal error (meaning the Parser stops that iteration of its run) while performing despawns - Our resource functionality was refering to some mailing client we used a long time ago that is no longer maintained. I have stripped that from our resource-related code now.

[Sobuno]

This might be as the job could be taking longer than one hour to complete - there are that many resources to check.

I just started a run by hand - It took about 30 seconds to complete for all servers, so this shouldn't be the issue.

[Sobuno]

the parser data seems to be 12 hours behind Galaxy Harvester. Also some spawns like the new phrik alum says its suppose to be available on talus and its not. Otherwise aside from 293 listed and SWGAide showing 291 and 12 hours off it seems to be mostly accurate. Anyone else noticing discrepancies?

Regarding Oimoium and Talus, I see in the history on GH that it was added to Talus there (https://galaxyharvester.net/resource.py/14/oimoium) and then cleaned up the next morning. There is no special behaviour relating to the individual planets and removing faulty markings - I'll try taking a quick look at that situation as it should be somewhat easily fixed.

Update: The Parser now removes availability for planets marked as available by itself under the assumption that it was a mistake from someone at GH.

[divory]

think it just canned again. All the yavin stuff just went unavailable....Only on the SWGAIde tool. There was an update pushed to the tool about an hour ago and it took 30 of the flora and creature resources and marked them as unavailable. I would have never noticed but i have a few harvesters (heavy flora) on the wild corn Kausnili. When i checked the website the resource is still available.
SWG AIDE shows 327 total resources available and the web site shows 358 available.
I was able to "revive" the yavin resources with SWGAIDE and the numbers are back to 357/358 Although the spawn timer was reset its at least accurate on both.

[Slyvampy]

the parser data seems to be 12 hours behind Galaxy Harvester.

This was as I changed the timer to be updated once every 12 hours as I changed it to see if it would solve the problem (see above posts). If someone at Galaxy Harvester or anyone can confirm how frequent they update their XML export we can change the timer to match this.