The selection operation in a database management system takes a table and a condition as input. It checks each row in the table to see if it meets the condition. If the row satisfies the condition, it is included in a new result table. If not, it is excluded. This process continues until all rows are checked. The original table is not changed. The output is a new table containing only the rows that meet the condition. For example, selecting employees with salary greater than 3000 will include only those employees whose salary is above 3000. This operation helps to find specific data from large tables easily.