The search algorithm is a list of rules used by search engines to rank matching results when a user performs a search. Google famously has 200+ ranking factors but nobody knows what all these ranking factors are.
See Google Algorithm for details.
Starter
See what people search and spy on competitors.
Webmaster Tools
Get Ahrefs data on your site and fix what matters.