Yes. You are required to pay your loan back in amortized amounts not less than quarterly for a period of time up to five years. A loan for the purchase of a residence can be paid back for a period of time up to 30 years. In addition if you have had a previous default on a loan you must pay the loan back from payroll deductions.
Your loan proceeds are provided from the investment provider and not withdrawn from your account. Your provider separately tracks the amount of the loan that is outstanding from you’re other account value. In the event you fail to pay on your loan you will default. If you default your investment provider will not be able to take a distribution from the account to cover the defaulted amount until you have met a triggering event.
The IRS considers the loan a distribution at the time it is taken and as such it is a taxable event, but the IRS waives taxation on the proceeds as long as these repayment conditions are met. If the loan is defaulted you are subject to income tax and possible early withdrawal penalties on the amount of proceeds outstanding at the time of the default.
In addition you will again be subject to income tax on any amount recovered by the investment company when they are able to recoup the defaulted amount at the time of a triggering event, such as reaching 59 ½ for a 403(b) or severance from employment.
If you pay back your loan you are not subject to taxation on the proceeds of the loan.