Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0007526Dwarf FortressAdventure Mode -- Sitespublic2014-07-20 17:122016-11-03 17:48
ReporterRobsoie 
Assigned ToDwarfu 
PrioritynormalSeverityminorReproducibilityalways
StatusconfirmedResolutionopen 
PlatformOSOS Version
Product Version0.40.04 
Target VersionFixed in Version 
Summary0007526: Dark towers contain thousands of goblins and trolls, causing lag
Descriptionon a Pocket World generated with 0.40.04, with only 50 years of history to make sure i still had population low (even despite the bug fixes about population, i wanted to keep things low for my new version test), i ran into a severe slowdown in adventure mode while inside of a goblin site.

At first when i entered the goblin territory (as Travel can't go in) , game was perfectly smooth and fast, i moved next to a couple of goblins tower that had some goblins on its top, still no slowdown, still fast.

Then as i was starting to approach the location of the main dark fortress, suddenly the game slowed down very noticably.
Here is the save just before it happens :
http://dffd.wimbli.com/file.php?id=9093 [^]

Load the save, at that point,the game is still fast and smooth, walk to the south , after several steps you'll encounter a very noticable slowdown.
It's painful but still you can relatively walk.
Continue to the south, there's a river, go past the river (jump/swim/etc..)

You'll arrive in front of a goblin tower you'll see the goblin line of sight signaling it as you're in stealth mode), on the top of it there are only a dozen of goblins.
Still walk to the south past the tower, and you'll arrive near the main dark fortress, and you'll notice the slowdown is really severe at that point
Additional InformationSaves:
http://dffd.bay12games.com/file.php?id=9093 [^] (0.40.04)
http://dffd.bay12games.com/file.php?id=9094 [^] (0.40.04)
http://dffd.bay12games.com/file.php?id=9184 [^] (0.40.05)
http://dffd.bay12games.com/file.php?id=9324 [^] (0.40.06)
http://dffd.bay12games.com/file.php?id=9513 [^] (0.40.10)
http://dffd.bay12games.com/file.php?id=9558 [^] (0.40.10)
http://dffd.bay12games.com/file.php?id=9715 [^] (0.40.12)
http://dffd.bay12games.com/file.php?id=10842 [^] (0.40.24)
http://dffd.bay12games.com/file.php?id=11867 [^] (0.42.06, 0009637, very large)
TagsNo tags attached.
Attached Files

- Relationships
related to 0007698new Excessive numbers of underground critters 
related to 0007547assignedFootkerchief TOTAL_CIV_POPULATION only affects historical figures, not abstracted populations 
related to 0007461new Unretired fort gets 1000+ goblin invaders 
has duplicate 0007578resolvedFootkerchief Serious FPS dropdown on enemies' sites 
has duplicate 0007878resolvedFootkerchief Dark pit population sizes make game unplayable 
has duplicate 0009177resolvedKnight Otu FPS death when approaching dark tower in adv mode 
has duplicate 0009170resolvedKnight Otu unplayable lag on some sites in adventure mode 
has duplicate 0009676resolvedDwarfu Troll population grows uncontrollably 
has duplicate 0009637resolvedlethosor assault on goblin city abruptly plagued by minutes-long processing time per turn 
related to 0008334new Trolls in goblin sites respawning constantly 
related to 0007009new Important slowdown near a goblin site 

-  Notes
(0027082)
Robsoie (reporter)
2014-07-20 17:21

Additional saved game , after i very very slowly and painfully managed to enter the fortress and explore it a bit to find where was the population , on the surface of the dark fortress only few trolls and few goblins, couldn't be what was the source of the slowdown, then after more exploring underground of that fortress i see a giant room with hundred of trolls there :

http://dffd.wimbli.com/file.php?id=9094 [^]
(0027756)
Robsoie (reporter)
2014-07-28 06:23
edited on: 2014-07-28 06:39

Still happening in 40.05 , i thought it would be fixed with the overpopulation fixes, but in that save :
http://dffd.wimbli.com/file.php?id=9184 [^]

I left the fortress and moved toward the dark fortress on the south east of the world, only to have the game being simply unplayable in term of slow and lag at that point.

I opened that 40.05 pocket world with World Viewer and it told me at the dark fortress site :
1 Dwarf (probably the guy i retired there)
1 Mouse Monster
4951 goblins !
5254 trolls !

No surprise unfortunately it is unplayable (and that's a pocket world)

(0027820)
Robsoie (reporter)
2014-07-28 15:37

In 40.05 i experimented today a lot with
MAX_STARTING_CIV_NUMBER
MAX_POP_NUMBER
MAX_SITE_POP_NUMBER
in the entity_default.txt
and
TOTAL_CIV_NUMBER
TOTAL_CIV_POPULATION
SITE_CAP
from worlgen.txt

And from my experimentations, i ended noticing that the goblins' dark fortresses always end having, in each of them, more than 1000 individuals of goblins (and as much trolls) inside, while every other civs (dwarves, humans, elves) have much more reasonnable 100/150 individuals for their main bigger site.
This when i set the exact same settings for each civs in the entity_default.txt

Even with only 20 years of history and no war (from what i see in World Viewer) , there's always that huge amount of goblins+trolls in a dark fortress, overwhelming numbers that just make the game unplayable when you're adventuring close to the site.
(0027823)
Dwarfu (manager)
2014-07-28 15:47

Thanks for the saves/follow-ups.
(0028518)
Borschwanger (reporter)
2014-08-07 06:37
edited on: 2014-08-07 06:39

The three most populous sites on my current world are actually elves this time. Elves total 8-11k with various animal populations ranging from a few thousand up to 15300 just for Tigers+Jaguars+Gorillas. Several Dark Fortresses follow up just behind. Year 250.

http://dffd.wimbli.com/file.php?id=9324 [^]

Sunsivy, Oaksilvers, Conflictfogs, Wheeledmalice, Plaguefield, Hexfrost, Balanceplague, Realmsterror, Waddlehate, and Beachedseduced are worst. A couple more in the low thousands and then it drops off to below 1000 in mostly caves and then quickly down to under 500. Seems about 90-95% of sites have roughly 200 or less residents.

In addition to the things Robsoie tried, I have also tried making worlds where Goblins and Elves were almost exclusively homosexual species by adding [orientation] tags to no effect (not in this world). Either it doesn't work like I imagine or I did it wrong (never did try 100% homosexual, though I probably should have...). Only mod is Phoebus and used some advanced worldgen parameters (savagery, good/evil, (semi)megabeasts, probably a few others).

(0028570)
smjjames (reporter)
2014-08-07 22:06

Sorry for offtopic, but is [orientation] tags really a thing?
(0028571)
Borschwanger (reporter)
2014-08-07 22:52

I dunno. It is in the wiki.

http://dwarffortresswiki.org/index.php/DF2014:Creature_token#ORIENTATION [^]

http://dwarffortresswiki.org/index.php/DF2014:Marriage [^]
(0028582)
Footkerchief (manager)
2014-08-08 09:00

0007905 pertains to ORIENTATION. Please keep the discussion here on topic.
(0029610)
Robsoie (reporter)
2014-08-25 07:37
edited on: 2014-08-25 07:43

world generated on 40.10 at only +/- 50 years , no raws edited out of the gob/elf/human/dwarves names files to remove accentuated characters
save there :
http://dffd.wimbli.com/file.php?id=9513 [^]

An elf bowman adventurer left his companion waiting at the entrance of the goblin site, all was well, until i started to approach 2 goblins dark towers on my way to their dark fortress, at such point it slowdown heavily.

But from the point of the save continue directly to the North , moving between the 2 dark towers, toward the dark fortress, it's going to be slower and slower into simply unplayable if you don't sprint (in which it's slow but at least you can move).

according to legends viewer, the dark fortress should have
1310 Goblins
802 Trolls
798 Ogres
5 Goblin Outcasts
1 Lizard Demons
1 Elf Outcasts

(0029777)
smjjames (reporter)
2014-08-29 12:19
edited on: 2014-08-29 12:25

I don't know if Toady One needs more saves to help solve this or not, but heres one of a tower site that is barely playable. I'm stubborn enough to just try and plow my way through it.

http://dffd.wimbli.com/file.php?id=9558 [^]

Nothing special about the save other than to demonstrate the laggyness. In the save, I'm just trying to wait for one of the rescued prisoners to snap out of it. Right now, I wish I could just pick up the kid as I move on or give him something to cheer him up and snap out of it because it's taking too long as is.

According to Legendsviewer, there should be roughly:

10154 Goblins
5100 Trolls
131 Humans
78 Human Prisoners
1 Skunk Fiends

I already killed some trolls and goblins, so it'll be a little less.

Also, my computer is a pretty good one, so, with computers not that good, it's unplayable.

(0029820)
smjjames (reporter)
2014-08-31 07:43

Need more savegames?
(0029913)
samanato (reporter)
2014-09-03 08:48

This seems to be related to the goblins' lack of need to eat or drink. Putting [NO_EAT] and [NO_DRINK] on kobolds, for example, will swell up cave populations to thousands in a similar way as in dark fortresses.
(0029915)
smjjames (reporter)
2014-09-03 09:15
edited on: 2014-09-03 09:44

They can still have thousands in a cave without those tags. I'll try removing those and see how things go.

Okay, in the world I tested it in, the largest populated tower has 2267 Goblins and 1850 Trolls, about what you would get with some of the most populated human towns/cities. Most of them have goblin and troll populations that range between 1k to 2k each, seeming to average between 1,200 and 1,500. Theres also several with each troll and goblin population less than a thousand.

Seems MUCH more reasonable to me.

To note, I did make them mortal with the same lifespans as dwarves (elves are still immortal because elves), but that didn't affect the populations.

Toady One, I'm sure you're reading this, what was the origional reason for putting the [NO_EAT] and [NO_DRINK] tags on goblins? Might not be needed anymore.

(0029916)
Quietust (reporter)
2014-09-03 09:53
edited on: 2014-09-03 09:59

In the original save linked in the bug report itself, there are 174 units in play at first, and after walking south and triggering the lag, there are 461 units in play. If I walk even further south to load more of the fortress (across the river and over the next trench), it increases further to 611.

smjjames's savegame has a whopping 1,606 units in play, which is clearly excessive.

(0029917)
smjjames (reporter)
2014-09-03 10:07
edited on: 2014-09-03 10:39

In this new world I generated for adv mode, the three largest towers (and the only three to breach 3k) have populations of 3198 Goblins 2250 Trolls and 3578 Goblins 2550 Trolls and 5497 Goblins 3100 Trolls. Which is MUCH better than the 5k+, sometimes up to 10k for each population. Most towers have less than 2k for each population and some have less than 1k for each.

Removing the [NO_EAT] and [NO_DRINK] tags help, but the way that the goblins are spread out, or rather, not spread out, causes lag problems and then theres the trolls doing crazy pathing which only adds to the issues.

Edit: Over a thousand in play, wow. I'll check out one of the towers in this modified save and see how thing are.

(0029921)
smjjames (reporter)
2014-09-03 15:46
edited on: 2014-09-03 17:30

Now testing a low pop tower with populations of
1275 Goblins
99 Humans
67 Human Prisoners
12 Dwarves
4 Dwarf Prisoners
1 Lemur Monsters

This goblin civ doesn't have trolls at that site for some reason, other sites of that civ have trolls though. I split off a compressed file from the spot I did the first loaded unit count if you guys request it.

Going eastward directly towards the spire. The first small tower I get to has 139 loaded units, 69 goblins, 36 crundles and the rest various cavern wildlife. No significant lag.

Second count, near the river before the spire, loaded unit number is 122. Mild conversation spam related lag and a bit of pathing lag.

Third count, approx center of spire, loaded unit number is 182, including cave wildlife. Slight amount of lag.

Second test with a Dark Tower population of:
3578 Goblins
2550 Trolls
104 Humans
25 Human Prisoners
3 Kobolds (I gave kobolds normal speech, which is why they are there)
3 Dwarves
1 Llama Fiends

First small tower reached and the lag is starting. Units loaded (or otherwise seen in DFhacks reveal+exterminate) is 396 with 226 goblins and 130 crundles, the rest, various cave critters plus a few humans.

Second reading, a couple tiles outside the tower and the lag has ramped up, loaded unit count is 690, with 468 trolls and 159 goblins. And um, when I did a wait one hour to clear out the reveal data, DF crashed. I'll check if that was a DFhack related thing or something else.

Also, I've been getting fairly regular brief freezes with DF as I move forward. Earlier I thought it was because of me using reveal all, but this time I didn't use it and I'm getting the brief lagspike pause. Also, I tested the wait one hour and while it didn't crash, it did briefly freeze, and I replicated what I did earlier when it crashed and it didn't that time, so, I don't know what that crash was about.

(0030216)
smjjames (reporter)
2014-09-16 08:47

Still happens in a 40.12 save which is modded to remove [NOEAT] and {NODRINK] from goblins plus givng them the lifespan of dwarves.

According to legendsviewer, the site pop is almost 1,600.

http://dffd.wimbli.com/file.php?id=9715 [^]
(0032666)
Witty (reporter)
2015-05-10 06:18

Still very much an issue in 40.24.

http://dffd.bay12games.com/file.php?id=10842 [^]

Vanilla goblins, only raw mod is allowing them to be adventure-playable.
The site lag makes the region entirely unplayable
(0033331)
Knight Otu (manager)
2015-12-03 03:22

Robsoie posted a save at 0009170.
(0034192)
Boatsniper (reporter)
2015-12-25 11:57

Still occurs in 42.03, most prevalent within Dark Goblin Fortresses.

I don't think it's normal for enormous Troll civilizations to exist in the dug out underground caverns (as in taking occupation titles, like weaver, administrator, animal trainer, etc.). To add onto the problem, the goblins and trolls will start killing each other once they get line of sight. It makes absolutely no sense.

This issue is easily reproducible, modded or otherwise; generate a world with a high site cap and decent history, adventure your way into a Dark Goblin Pit/Fortress, and watch the game become disastrously unplayable.
(0035724)
FantasticDorf (reporter)
2016-07-30 03:59
edited on: 2016-07-30 07:05

A quick check with a third party tool (legends reader found here: http://www.bay12forums.com/smf/index.php?topic=72702.msg6658295#msg6658295 [^] ) reveals that since 43.05 site populations are still exploding into enormous numbers. On my unmodded world there are globally 95200 goblins for a world with only 25347 historical figures to have ever lived, not to mention about ten thousand elves and 8000 humans.

Only 24375 goblins can be supported in the 195 dark pits that have been built to accommodate the sprawl which stays within the rules of 125 per site. With in my most prominent tower, it has 10851 goblins and 2526 trolls & 2526 beak dogs. The oldest and first tower but just 1 tower out of a network of towers with large similar statistics, and this trends between the 3 goblin civilisations to reach the big 95200 global number.

The wars that occur, such as my largest and latest war 'The War of Murders' has killed 23394 people, more than the 14491 historical figures that are supposed to be dead and the war still hasn't ended or either side feeling inclination to stop for more than 60 years (declared in 167, ongoing in 250)

Auxiliary populations are crashing the game and making it strenuous on systems in adventure mode to approach sites because livestock is spinning out of control to keep up with the numbers in order to 'feed' those who even don't eat, all these people and the subsequent pathing causes as described lag spikes.

Murders and war cannot keep on top of the goblins and i cannot fathom where or why all these unaccounted non-historical people are coming from.

@boatsniper trolls have always (intentionally or not) been like that, since they are civilians but also livestock they get the best of both worlds with automatic job assignment and population scaling. However they cannot commit to a job because they are slow learners without 3rd party tool exploitation.

(0035728)
Murphy (reporter)
2016-07-31 03:10

I thought they wouldn't ALL load at the same time? 1000+ goblins is the population pool, and individual non-historical goblins are pulled out when you visit the tower, and then they offload when you leave. You shouldn't actually ever see all 1000 present at the same time.

Might not be the case with trolls, however, since they count as animals.

I've noticed that the structure of dark fortress can link cavern layers together, causing water from the above layer to fall through the holes into the layer below. Could that be the real reason of the lag?
(0035735)
LordKnows (reporter)
2016-08-02 03:10
edited on: 2016-08-02 10:16

My 43.05 world seems affected by a similar, but somewhat different issue of overpopulation, and I was advised to take my save here.

http://dffd.bay12games.com/file.php?id=12322 [^]

Notable about this is that this only happened AFTER world-gen had finished and I started playing fortress and adventurers. In 850 there were around 2500 fly-men in the world, now there are over half a million largely located in just two sites, Brassdark and Pointrose. I had two consecutive crashes on attempting to enter pointrose earlier, but just now was able to enter with only extreme performance issues (unsurprisingly, it's full of fly-men, 250 thousand apparently.)

I don't have a backup save before the population started growing unfortunately, but do have the xml dump from 850 which I read with legendsviewer to see the huge difference over the two years.

EDIT: I just noticed that the world and fly-man population had changed in the two weeks or so of in-game time that had passed since I last looked at the legends. Pointrose now only has 14629 fly men living there, which explains why it stopped crashing for me probably.

It seems that fly-man outcasts, of which the vast majority of Pointrose consisted of have simply disappeared or something leaving only the ca. 15k that were "normal" fly-men in Pointrose, and brassdark still has hundreds of thousands... Really strange.

Here is a backed up save before the outcasts disappeared, this is only two weeks before the save with a quarter-million less fly-man outcasts:
http://dffd.bay12games.com/file.php?id=12324 [^]


- Issue History
Date Modified Username Field Change
2014-07-20 17:12 Robsoie New Issue
2014-07-20 17:21 Robsoie Note Added: 0027082
2014-07-28 06:23 Robsoie Note Added: 0027756
2014-07-28 06:24 Robsoie Note Edited: 0027756 View Revisions
2014-07-28 06:39 Robsoie Note Edited: 0027756 View Revisions
2014-07-28 15:37 Robsoie Note Added: 0027820
2014-07-28 15:46 Dwarfu Relationship added related to 0007698
2014-07-28 15:47 Dwarfu Note Added: 0027823
2014-07-28 15:47 Dwarfu Assigned To => Dwarfu
2014-07-28 15:47 Dwarfu Status new => acknowledged
2014-07-28 15:48 Dwarfu Relationship added related to 0007547
2014-07-28 15:51 Dwarfu Relationship added related to 0007461
2014-08-06 18:03 Footkerchief Summary heavy slowdown case near dark fortress => Dark towers contain thousands of goblins and trolls, causing lag
2014-08-06 18:04 Footkerchief Relationship added has duplicate 0007578
2014-08-06 18:04 Footkerchief Issue Monitored: alexis06
2014-08-06 18:04 Footkerchief Relationship added has duplicate 0007878
2014-08-06 21:07 Talvieno Issue Monitored: Talvieno
2014-08-07 06:37 Borschwanger Note Added: 0028518
2014-08-07 06:38 Borschwanger Note Edited: 0028518 View Revisions
2014-08-07 06:39 Borschwanger Note Edited: 0028518 View Revisions
2014-08-07 21:57 Footkerchief Status acknowledged => confirmed
2014-08-07 22:06 smjjames Note Added: 0028570
2014-08-07 22:52 Borschwanger Note Added: 0028571
2014-08-08 07:11 samanato Issue Monitored: samanato
2014-08-08 09:00 Footkerchief Note Added: 0028582
2014-08-25 07:37 Robsoie Note Added: 0029610
2014-08-25 07:43 Robsoie Note Edited: 0029610 View Revisions
2014-08-29 12:19 smjjames Note Added: 0029777
2014-08-29 12:25 smjjames Note Edited: 0029777 View Revisions
2014-08-29 12:25 smjjames Note Edited: 0029777 View Revisions
2014-08-31 07:43 smjjames Note Added: 0029820
2014-09-03 08:48 samanato Note Added: 0029913
2014-09-03 09:15 smjjames Note Added: 0029915
2014-09-03 09:36 smjjames Note Edited: 0029915 View Revisions
2014-09-03 09:38 smjjames Note Edited: 0029915 View Revisions
2014-09-03 09:44 smjjames Note Edited: 0029915 View Revisions
2014-09-03 09:53 Quietust Note Added: 0029916
2014-09-03 09:54 Quietust Note Edited: 0029916 View Revisions
2014-09-03 09:59 Quietust Note Edited: 0029916 View Revisions
2014-09-03 10:07 smjjames Note Added: 0029917
2014-09-03 10:08 smjjames Note Edited: 0029917 View Revisions
2014-09-03 10:35 4maskwolf Issue Monitored: 4maskwolf
2014-09-03 10:39 smjjames Note Edited: 0029917 View Revisions
2014-09-03 15:46 smjjames Note Added: 0029921
2014-09-03 15:56 smjjames Note Edited: 0029921 View Revisions
2014-09-03 16:03 smjjames Note Edited: 0029921 View Revisions
2014-09-03 16:27 smjjames Note Edited: 0029921 View Revisions
2014-09-03 16:53 smjjames Note Edited: 0029921 View Revisions
2014-09-03 17:03 smjjames Note Edited: 0029921 View Revisions
2014-09-03 17:27 smjjames Note Edited: 0029921 View Revisions
2014-09-03 17:30 smjjames Note Edited: 0029921 View Revisions
2014-09-04 16:05 Saram-61-97-kon Issue Monitored: Saram-61-97-kon
2014-09-16 08:47 smjjames Note Added: 0030216
2014-09-22 09:50 Footkerchief Relationship added related to 0008334
2014-10-12 07:34 NekoHatsu Issue Monitored: NekoHatsu
2015-05-10 06:18 Witty Note Added: 0032666
2015-05-10 10:15 lethosor Additional Information Updated View Revisions
2015-05-10 10:16 lethosor Sticky Issue No => Yes
2015-12-03 03:19 Knight Otu Relationship added has duplicate 0009177
2015-12-03 03:20 Knight Otu Relationship added has duplicate 0009170
2015-12-03 03:20 Knight Otu Relationship added related to 0007009
2015-12-03 03:22 Knight Otu Note Added: 0033331
2015-12-25 11:57 Boatsniper Note Added: 0034192
2016-02-09 02:02 park66665 Issue Monitored: park66665
2016-05-08 17:08 Dwarfu Relationship added has duplicate 0009676
2016-07-28 19:11 lethosor Relationship added has duplicate 0009637
2016-07-28 19:12 lethosor Additional Information Updated View Revisions
2016-07-30 03:59 FantasticDorf Note Added: 0035724
2016-07-30 07:05 FantasticDorf Note Edited: 0035724 View Revisions
2016-07-31 03:10 Murphy Note Added: 0035728
2016-08-02 03:10 LordKnows Note Added: 0035735
2016-08-02 10:16 LordKnows Note Edited: 0035735 View Revisions
2016-11-03 17:48 FantasticDorf Note Added: 0036031
2017-01-28 14:14 FantasticDorf Note Deleted: 0036031


Copyright © 2000 - 2010 MantisBT Group
Powered by Mantis Bugtracker