Protecting the data directory with .htaccess?

Post

Posted
Rating:
#5745 (In Topic #1344)
Avatar
Standard member
ironfeather is in the usergroup ‘Well-settled’

google console is complaining about Server error (5xx)

Hello,

Received a "New Coverage issue detected for site https://nandalow.com/"  warning email from google search console. Apparently they are scanning my https://nandalow.com/data/ directory and getting some 5xx type errors….. 

For example:

https://compo.sr/data/notifications.php



google is complaining about the data/notifications.php….  Would you recommend we add to the disallow list on the robots.txt file?
 

The robots.txt does not specifically block the /data/ folder (I dont think):

https://compo.sr/robots.txt

the .htaccess file in the /data/ folder is empty

Thanks for your thoughts
 

 

Last edit: by ironfeather


———–
Publisher of IronFeather Journal since 1987.  Host of KGNU Colorado Radio for 20 years. 
Currently in Japan & decided to focus on Composr as my number one CMS.
Composr site for community of Hokkaido:  Nandalow.com
Composr site for my freelance work: Futurecode.jp
My Compsr edits : 
http://ironfeather.com/bbs/viewtopic.php?f=12&t=2862
Twitter: https://twitter.com/futurecodejp

 
Online now: No Back to the top

Post

Posted
Rating:
#5746
Avatar
Site director
Chris Graham is in the usergroup ‘Administrators’
Hi,

Our approach is to put out X-Robots-Tag headers, or <meta> declarations, telling search engines if they should not be indexing something. That way people don't need to worry about robots.txt for anything inbuilt to Composr.

In this particular case if a required parameter is missing for notifications.php it doesn't get to the point of putting out that header.
Fixed in:
Allow No-Robots header to work even on corrupt notification URL · ocproducts/composr@8f939dd · GitHub


Become a fan of Composr on Facebook or add me as a friend. Add me on on Twitter. Follow me on Minds (where I am most active). Support me on Patreon

Was I helpful?
  • If not, please let us know how we can do better (please try and propose any bigger ideas in such a way that they are fundable and scalable).
  • If so, please let others know about Composr whenever you see the opportunity or support me on Patreon.
  • If my reply is too Vulcan or expressed too much in business-strategy terms, and not particularly personal, I apologise. As a company & project maintainer, time is very limited to me, so usually when I write a reply I try and make it generic advice to all readers. I'm also naturally a joined-up thinker, so I always express my thoughts in combined business and technical terms. I recognise not everyone likes that, don't let my Vulcan-thinking stop you enjoying Composr on fun personal projects.
  • If my response can inspire a community tutorial, that's a great way of giving back to the project as a user.
Online now: No Back to the top
1 guest and 0 members have just viewed this.

Statistics

Users online:

mytracker, Salman, Philip, babu, MVLipwig, Vaiva, gabriel58, Paul D, amit.nigam

Forum statistics:
  • 1,086 topics, 5,242 posts, 6,117 members
  • Our newest member is stitchmax
Birthdays:
Back to Top