How to index intranet with .net domains

YaCy
1

I have fresh yacy installation, want to use it for local use only, my intranet uses .net
when i try to index anything, for example mediawiki i get this
“Crawling of “twiki.net” failed. Reason: denied_(the host ‘twiki.net’ is global, but global addresses are not accepted: 192.168.7.17)/”

it did reach the right host indeed, but how do i make yacy accept my ip or domains as local addresses

i did some googling and didnt reach anything

thanks!

2

well, after more reading i can understand that i have to use network.unit.domainlist
i created file called searchme.txt in yacy root dir, add to it

twiki.net
http://twiki.net

edited defaults/yacy.network.intranet.unit and added

network.unit.domainlist = seachme.txt

i still cant index twiki.net

3

i got ignored i think :frowning:

just how do i format the domainlist, just put the links there? do i need to add something before each line?
YaCy seems to access my domainlist file, but not able to understand it i think?

I tried to make the host file accessible over http, and i get that the file got accessed
192.168.7.146 - - [05/Jun/2020 14:02:27] "GET /allowedsites.txt HTTP/1.1" 200

but still get in YaCy that its not in the domainlist :confused:

4

YaCy recognizes internet/intranet ranges using the IP after DNS resolving. So if you are using .net TLDs within your intranet, then you must switch to intranet mode.

YaCy differs strictly between internet and intranet indexing because this is a privacy issue.

5

I am using yacy as an intranet search engine and I get " Crawling of “http://devuan.fritz.box” failed. Reason: denied_(the host ‘devuan.fritz.box’ is global, but global addresses are not accepted: 127.0.0.1)/".
imho the tld .box isn’t global nor the IP.
I started a crawl with this domain after starting yacy for the first time and later I added devuan.fritz.box to /etc/hosts. Later I stopped yacy, moved the directory to another partition and started over. The running crawl wasn’t executed like before. Now I got the error like above and wasn’t able to start a crawl again.

The background: I want to have a server with kiwix, dwww and search engine and want to access it from other local devices. Therefore I can’t run the crawl with 127.0.0.1. Have a look at https://friendsofdevuan.org/doku.php/user:hum/self-reliance to see more details.

6

Using devuan instead of devuan.fritz.box works for me.